Running make with -j24 GOPATH set to /home/ajwerner/go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /home/ajwerner/go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /home/ajwerner/go Cleaning old generated files. # The bootstrap target creates, among other things, ./bin/stress. build/builder.sh make bin/.bootstrap Running make with -j24 GOPATH set to /go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /go Cleaning old generated files. Detected change in build system. Rebooting Make. # Special case for 'prefix' variable, which is required by Homebrew. Running make with -j24 GOPATH set to /go Cleaning old generated files. make: 'bin/.bootstrap' is up to date. build/builder.sh make test GOFLAGS=" -v -c -o partitionccl.test" PKG=./pkg/ccl/partitionccl Running make with -j24 GOPATH set to /go Cleaning old generated files. make[1]: Nothing to be done for 'build_lib_static'. GOFLAGS= go test -v -c -o partitionccl.test -tags ' gss make x86_64_pc_linux_gnu' -ldflags '-X github.com/cockroachdb/cockroach/pkg/build.typ=development -extldflags "" -X "github.com/cockroachdb/cockroach/pkg/build.tag=v19.2.0-alpha.20190606-1964-g982579c" -X "github.com/cockroachdb/cockroach/pkg/build.rev=982579c72d1925475e93df1a783f48c3911fd2da" -X "github.com/cockroachdb/cockroach/pkg/build.cgoTargetTriple=x86_64-pc-linux-gnu" ' -run "." -timeout 12m ./pkg/ccl/partitionccl ajwerner-stress: putting (dist) bin.docker_amd64/stress stress . 1: done 2: done 3: done 4: done ajwerner-stress: putting (dist) ./partitionccl.test partitionccl.test .......... 1: done 2: done 3: done 4: done 0 runs so far, 0 failures, over 5s 0 runs so far, 0 failures, over 10s 0 runs so far, 0 failures, over 15s 0 runs so far, 0 failures, over 20s 0 runs so far, 0 failures, over 25s 0 runs so far, 0 failures, over 30s 0 runs so far, 0 failures, over 35s 0 runs so far, 0 failures, over 40s 0 runs so far, 0 failures, over 45s 0 runs so far, 0 failures, over 50s 0 runs so far, 0 failures, over 55s 0 runs so far, 0 failures, over 1m0s 0 runs so far, 0 failures, over 1m5s 0 runs so far, 0 failures, over 1m10s 3 runs so far, 0 failures, over 1m15s 18 runs so far, 0 failures, over 1m20s 45 runs so far, 0 failures, over 1m25s 58 runs so far, 0 failures, over 1m30s 64 runs so far, 0 failures, over 1m35s 64 runs so far, 0 failures, over 1m40s 64 runs so far, 0 failures, over 1m45s 64 runs so far, 0 failures, over 1m50s 64 runs so far, 0 failures, over 1m55s 64 runs so far, 0 failures, over 2m0s 64 runs so far, 0 failures, over 2m5s 64 runs so far, 0 failures, over 2m10s 64 runs so far, 0 failures, over 2m15s 64 runs so far, 0 failures, over 2m20s 64 runs so far, 0 failures, over 2m25s 73 runs so far, 0 failures, over 2m30s 84 runs so far, 0 failures, over 2m35s 100 runs so far, 0 failures, over 2m40s 116 runs so far, 0 failures, over 2m45s 124 runs so far, 0 failures, over 2m50s 127 runs so far, 0 failures, over 2m55s 128 runs so far, 0 failures, over 3m0s 128 runs so far, 0 failures, over 3m5s 128 runs so far, 0 failures, over 3m10s 128 runs so far, 0 failures, over 3m15s 128 runs so far, 0 failures, over 3m20s 128 runs so far, 0 failures, over 3m25s 128 runs so far, 0 failures, over 3m30s 128 runs so far, 0 failures, over 3m35s 132 runs so far, 0 failures, over 3m40s 139 runs so far, 0 failures, over 3m45s 151 runs so far, 0 failures, over 3m50s 167 runs so far, 0 failures, over 3m55s 176 runs so far, 0 failures, over 4m0s 184 runs so far, 0 failures, over 4m5s 190 runs so far, 0 failures, over 4m10s 191 runs so far, 0 failures, over 4m15s 191 runs so far, 0 failures, over 4m20s 192 runs so far, 0 failures, over 4m25s 192 runs so far, 0 failures, over 4m30s 192 runs so far, 0 failures, over 4m35s 192 runs so far, 0 failures, over 4m40s 192 runs so far, 0 failures, over 4m45s 194 runs so far, 0 failures, over 4m50s 197 runs so far, 0 failures, over 4m55s 206 runs so far, 0 failures, over 5m0s 219 runs so far, 0 failures, over 5m5s 230 runs so far, 0 failures, over 5m10s 238 runs so far, 0 failures, over 5m15s 247 runs so far, 0 failures, over 5m20s 255 runs so far, 0 failures, over 5m25s 255 runs so far, 0 failures, over 5m30s 256 runs so far, 0 failures, over 5m35s 256 runs so far, 0 failures, over 5m40s 256 runs so far, 0 failures, over 5m45s 256 runs so far, 0 failures, over 5m50s 256 runs so far, 0 failures, over 5m55s 256 runs so far, 0 failures, over 6m0s 259 runs so far, 0 failures, over 6m5s 265 runs so far, 0 failures, over 6m10s 274 runs so far, 0 failures, over 6m15s 285 runs so far, 0 failures, over 6m20s 292 runs so far, 0 failures, over 6m25s 299 runs so far, 0 failures, over 6m30s 308 runs so far, 0 failures, over 6m35s 316 runs so far, 0 failures, over 6m40s 317 runs so far, 0 failures, over 6m45s 320 runs so far, 0 failures, over 6m50s 320 runs so far, 0 failures, over 6m55s 320 runs so far, 0 failures, over 7m0s 320 runs so far, 0 failures, over 7m5s 320 runs so far, 0 failures, over 7m10s 322 runs so far, 0 failures, over 7m15s 327 runs so far, 0 failures, over 7m20s 332 runs so far, 0 failures, over 7m25s 338 runs so far, 0 failures, over 7m30s 345 runs so far, 0 failures, over 7m35s 355 runs so far, 0 failures, over 7m40s 366 runs so far, 0 failures, over 7m45s 373 runs so far, 0 failures, over 7m50s 378 runs so far, 0 failures, over 7m55s 380 runs so far, 0 failures, over 8m0s 383 runs so far, 0 failures, over 8m5s 383 runs so far, 0 failures, over 8m10s 384 runs so far, 0 failures, over 8m15s 384 runs so far, 0 failures, over 8m20s 386 runs so far, 0 failures, over 8m25s 388 runs so far, 0 failures, over 8m30s 392 runs so far, 0 failures, over 8m35s 398 runs so far, 0 failures, over 8m40s 401 runs so far, 0 failures, over 8m45s 407 runs so far, 0 failures, over 8m50s 419 runs so far, 0 failures, over 8m55s 426 runs so far, 0 failures, over 9m0s 436 runs so far, 0 failures, over 9m5s 439 runs so far, 0 failures, over 9m10s 443 runs so far, 0 failures, over 9m15s 445 runs so far, 0 failures, over 9m20s 447 runs so far, 0 failures, over 9m25s 447 runs so far, 0 failures, over 9m30s 449 runs so far, 0 failures, over 9m35s 451 runs so far, 0 failures, over 9m40s 456 runs so far, 0 failures, over 9m45s 460 runs so far, 0 failures, over 9m50s 462 runs so far, 0 failures, over 9m55s 468 runs so far, 0 failures, over 10m0s 475 runs so far, 0 failures, over 10m5s 482 runs so far, 0 failures, over 10m10s 492 runs so far, 0 failures, over 10m15s 497 runs so far, 0 failures, over 10m20s 503 runs so far, 0 failures, over 10m25s 508 runs so far, 0 failures, over 10m30s 510 runs so far, 0 failures, over 10m35s 511 runs so far, 0 failures, over 10m40s 513 runs so far, 0 failures, over 10m45s 514 runs so far, 0 failures, over 10m50s 516 runs so far, 0 failures, over 10m55s 518 runs so far, 0 failures, over 11m0s 524 runs so far, 0 failures, over 11m5s 529 runs so far, 0 failures, over 11m10s 533 runs so far, 0 failures, over 11m15s 544 runs so far, 0 failures, over 11m20s 547 runs so far, 0 failures, over 11m25s 554 runs so far, 0 failures, over 11m30s 561 runs so far, 0 failures, over 11m35s 565 runs so far, 0 failures, over 11m40s 570 runs so far, 0 failures, over 11m45s 573 runs so far, 0 failures, over 11m50s 575 runs so far, 0 failures, over 11m55s 577 runs so far, 0 failures, over 12m0s 578 runs so far, 0 failures, over 12m5s 581 runs so far, 0 failures, over 12m10s 584 runs so far, 0 failures, over 12m15s 589 runs so far, 0 failures, over 12m20s 594 runs so far, 0 failures, over 12m25s 602 runs so far, 0 failures, over 12m30s 605 runs so far, 0 failures, over 12m35s 610 runs so far, 0 failures, over 12m40s 616 runs so far, 0 failures, over 12m45s 623 runs so far, 0 failures, over 12m50s 629 runs so far, 0 failures, over 12m55s 632 runs so far, 0 failures, over 13m0s 636 runs so far, 0 failures, over 13m5s 637 runs so far, 0 failures, over 13m10s 641 runs so far, 0 failures, over 13m15s 643 runs so far, 0 failures, over 13m20s 643 runs so far, 0 failures, over 13m25s 647 runs so far, 0 failures, over 13m30s 654 runs so far, 0 failures, over 13m35s 658 runs so far, 0 failures, over 13m40s 669 runs so far, 0 failures, over 13m45s 670 runs so far, 0 failures, over 13m50s 673 runs so far, 0 failures, over 13m55s 683 runs so far, 0 failures, over 14m0s 688 runs so far, 0 failures, over 14m5s 692 runs so far, 0 failures, over 14m10s 695 runs so far, 0 failures, over 14m15s 699 runs so far, 0 failures, over 14m20s 704 runs so far, 0 failures, over 14m25s 705 runs so far, 0 failures, over 14m30s 708 runs so far, 0 failures, over 14m35s 711 runs so far, 0 failures, over 14m40s 714 runs so far, 0 failures, over 14m45s 720 runs so far, 0 failures, over 14m50s 723 runs so far, 0 failures, over 14m55s 731 runs so far, 0 failures, over 15m0s 735 runs so far, 0 failures, over 15m5s 737 runs so far, 0 failures, over 15m10s 744 runs so far, 0 failures, over 15m15s 753 runs so far, 0 failures, over 15m20s 757 runs so far, 0 failures, over 15m25s 761 runs so far, 0 failures, over 15m30s 766 runs so far, 0 failures, over 15m35s 768 runs so far, 0 failures, over 15m40s 771 runs so far, 0 failures, over 15m45s 772 runs so far, 0 failures, over 15m50s 776 runs so far, 0 failures, over 15m55s 780 runs so far, 0 failures, over 16m0s 783 runs so far, 0 failures, over 16m5s 788 runs so far, 0 failures, over 16m10s 797 runs so far, 0 failures, over 16m15s 798 runs so far, 0 failures, over 16m20s 802 runs so far, 0 failures, over 16m25s 812 runs so far, 0 failures, over 16m30s 816 runs so far, 0 failures, over 16m35s 820 runs so far, 0 failures, over 16m40s 826 runs so far, 0 failures, over 16m45s 829 runs so far, 0 failures, over 16m50s 834 runs so far, 0 failures, over 16m55s 835 runs so far, 0 failures, over 17m0s 839 runs so far, 0 failures, over 17m5s 841 runs so far, 0 failures, over 17m10s 844 runs so far, 0 failures, over 17m15s 851 runs so far, 0 failures, over 17m20s 855 runs so far, 0 failures, over 17m25s 860 runs so far, 0 failures, over 17m30s 865 runs so far, 0 failures, over 17m35s 866 runs so far, 0 failures, over 17m40s 870 runs so far, 0 failures, over 17m45s 881 runs so far, 0 failures, over 17m50s 886 runs so far, 0 failures, over 17m55s 888 runs so far, 0 failures, over 18m0s 895 runs so far, 0 failures, over 18m5s 899 runs so far, 0 failures, over 18m10s 902 runs so far, 0 failures, over 18m15s 903 runs so far, 0 failures, over 18m20s 906 runs so far, 0 failures, over 18m25s 909 runs so far, 0 failures, over 18m30s 914 runs so far, 0 failures, over 18m35s 917 runs so far, 0 failures, over 18m40s 924 runs so far, 0 failures, over 18m45s 928 runs so far, 0 failures, over 18m50s 931 runs so far, 0 failures, over 18m55s 935 runs so far, 0 failures, over 19m0s 941 runs so far, 0 failures, over 19m5s 948 runs so far, 0 failures, over 19m10s 953 runs so far, 0 failures, over 19m15s 960 runs so far, 0 failures, over 19m20s 963 runs so far, 0 failures, over 19m25s 965 runs so far, 0 failures, over 19m30s 970 runs so far, 0 failures, over 19m35s 972 runs so far, 0 failures, over 19m40s 976 runs so far, 0 failures, over 19m45s 980 runs so far, 0 failures, over 19m50s 984 runs so far, 0 failures, over 19m55s 988 runs so far, 0 failures, over 20m0s 993 runs so far, 0 failures, over 20m5s 997 runs so far, 0 failures, over 20m10s 1001 runs so far, 0 failures, over 20m15s 1009 runs so far, 0 failures, over 20m20s 1011 runs so far, 0 failures, over 20m25s 1018 runs so far, 0 failures, over 20m30s 1025 runs so far, 0 failures, over 20m35s 1027 runs so far, 0 failures, over 20m40s 1029 runs so far, 0 failures, over 20m45s 1034 runs so far, 0 failures, over 20m50s 1038 runs so far, 0 failures, over 20m55s 1042 runs so far, 0 failures, over 21m0s 1045 runs so far, 0 failures, over 21m5s 1050 runs so far, 0 failures, over 21m10s 1053 runs so far, 0 failures, over 21m15s 1056 runs so far, 0 failures, over 21m20s 1062 runs so far, 0 failures, over 21m25s 1063 runs so far, 0 failures, over 21m30s 1068 runs so far, 0 failures, over 21m35s 1073 runs so far, 0 failures, over 21m40s 1084 runs so far, 0 failures, over 21m45s 1087 runs so far, 0 failures, over 21m50s 1090 runs so far, 0 failures, over 21m55s 1093 runs so far, 0 failures, over 22m0s 1097 runs so far, 0 failures, over 22m5s 1100 runs so far, 0 failures, over 22m10s 1105 runs so far, 0 failures, over 22m15s 1108 runs so far, 0 failures, over 22m20s 1114 runs so far, 0 failures, over 22m25s 1118 runs so far, 0 failures, over 22m30s 1121 runs so far, 0 failures, over 22m35s 1126 runs so far, 0 failures, over 22m40s 1129 runs so far, 0 failures, over 22m45s 1134 runs so far, 0 failures, over 22m50s 1140 runs so far, 0 failures, over 22m55s 1147 runs so far, 0 failures, over 23m0s 1150 runs so far, 0 failures, over 23m5s 1155 runs so far, 0 failures, over 23m10s 1157 runs so far, 0 failures, over 23m15s 1161 runs so far, 0 failures, over 23m20s 1164 runs so far, 0 failures, over 23m25s 1167 runs so far, 0 failures, over 23m30s 1172 runs so far, 0 failures, over 23m35s 1178 runs so far, 0 failures, over 23m40s 1183 runs so far, 0 failures, over 23m45s shutting down I190903 20:12:50.559240 1 rand.go:85 Random seed: -9007629993438222398 === RUN TestRepartitioning I190903 20:12:50.584395 10 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 20:12:50.584447 10 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node W190903 20:12:50.625838 10 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 20:12:50.647044 10 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 20:12:50.656952 10 server/config.go:502 [n?] 1 storage engine initialized I190903 20:12:50.656977 10 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 20:12:50.656987 10 server/config.go:505 [n?] store 0: in-memory, size 0 B I190903 20:12:50.657134 10 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 20:12:50.657179 10 server/server.go:1369 [n?] **** add additional nodes by specifying --join=127.0.0.1:43339 I190903 20:12:50.660814 10 server/node.go:317 [n?] **** cluster b736962c-f08c-476e-9b06-40a804863214 has been created I190903 20:12:50.663275 10 gossip/gossip.go:394 [n1] NodeDescriptor set to node_id:1 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567541570662132240 cluster_name:"" sql_address: W190903 20:12:50.712827 270 storage/store.go:1546 [n1,s1,r6/1:/Table/{SystemCon…-11}] could not gossip system config: [NotLeaseHolderError] r6: replica (n1,s1):1 not lease holder; lease holder unknown I190903 20:12:50.714034 10 server/node.go:422 [n1] initialized store [n1,s1]: disk (capacity=512 MiB, available=512 MiB, used=0 B, logicalBytes=8.7 KiB), ranges=20, leases=0, queries=0.00, writes=0.00, bytesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=929.00 pMax=7848.00}, writesPerReplica={p10=0.00 p25=0.00 p50=0.00 p75=0.00 p90=0.00 pMax=0.00} I190903 20:12:50.714146 10 storage/stores.go:240 [n1] read 0 node addresses from persistent storage I190903 20:12:50.714275 10 server/node.go:636 [n1] connecting to gossip network to verify cluster ID... I190903 20:12:50.714310 10 server/node.go:656 [n1] node connected via gossip and verified as part of cluster "b736962c-f08c-476e-9b06-40a804863214" I190903 20:12:50.714480 10 server/node.go:503 [n1] node=1: started with [n1=] engine(s) and attributes [] I190903 20:12:50.714781 10 server/server.go:1989 [n1] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 20:12:50.714871 10 server/server.go:1486 [n1] starting https server at 127.0.0.1:41489 (use: 127.0.0.1:41489) I190903 20:12:50.714942 10 server/server.go:1491 [n1] starting postgres server at 127.0.0.1:46319 (use: 127.0.0.1:46319) I190903 20:12:50.714983 10 server/server.go:1493 [n1] starting grpc server at 127.0.0.1:43339 I190903 20:12:50.715023 10 server/server.go:1494 [n1] advertising CockroachDB node at 127.0.0.1:43339 I190903 20:12:50.810329 361 sql/event_log.go:130 [n1,intExec=optInToDiagnosticsStatReporting] Event: "set_cluster_setting", target: 0, info: {SettingName:diagnostics.reporting.enabled Value:true User:root} I190903 20:12:50.878850 335 sql/event_log.go:130 [n1,intExec=set-setting] Event: "set_cluster_setting", target: 0, info: {SettingName:version Value:19.1-9 User:root} I190903 20:12:50.886363 630 sql/event_log.go:130 [n1,intExec=disableNetTrace] Event: "set_cluster_setting", target: 0, info: {SettingName:trace.debug.enable Value:false User:root} I190903 20:12:50.928499 852 sql/event_log.go:130 [n1,intExec=initializeClusterSecret] Event: "set_cluster_setting", target: 0, info: {SettingName:cluster.secret Value:e90a49a5-215b-4c3b-848c-6983cf0335ac User:root} I190903 20:12:50.943577 932 sql/event_log.go:130 [n1,intExec=create-default-db] Event: "create_database", target: 50, info: {DatabaseName:defaultdb Statement:CREATE DATABASE IF NOT EXISTS defaultdb User:root} I190903 20:12:50.949954 460 sql/event_log.go:130 [n1,intExec=create-default-db] Event: "create_database", target: 51, info: {DatabaseName:postgres Statement:CREATE DATABASE IF NOT EXISTS postgres User:root} I190903 20:12:51.021015 10 server/server.go:1562 [n1] done ensuring all necessary migrations have run I190903 20:12:51.021081 10 server/server.go:1800 [n1] serving sql connections I190903 20:12:51.026699 1205 sql/event_log.go:130 [n1] Event: "node_join", target: 1, info: {Descriptor:{NodeID:1 Address:127.0.0.1:43339 Attrs: Locality:region=test,dc=dc1 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567541570662132240 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:46319} ClusterID:b736962c-f08c-476e-9b06-40a804863214 StartedAt:1567541570662132240 LastUp:1567541570662132240} I190903 20:12:51.043780 1203 server/server_update.go:68 [n1] no need to upgrade, cluster already at the newest version I190903 20:12:51.058346 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 20:12:51.098294 10 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 20:12:51.098316 10 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node W190903 20:12:51.124746 10 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 20:12:51.218402 10 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 20:12:51.253559 10 server/config.go:502 [n?] 1 storage engine initialized I190903 20:12:51.253587 10 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 20:12:51.253598 10 server/config.go:505 [n?] store 0: in-memory, size 0 B W190903 20:12:51.253696 10 gossip/gossip.go:1517 [n?] no incoming or outgoing connections I190903 20:12:51.253763 10 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 20:12:51.274333 1376 gossip/client.go:124 [n?] started gossip client to 127.0.0.1:43339 I190903 20:12:51.274724 1541 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:33569 I190903 20:12:51.275938 10 server/node.go:636 [n?] connecting to gossip network to verify cluster ID... I190903 20:12:51.275970 10 server/node.go:656 [n?] node connected via gossip and verified as part of cluster "b736962c-f08c-476e-9b06-40a804863214" I190903 20:12:51.277523 1541 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:33569 I190903 20:12:51.316677 10 server/node.go:372 [n?] new node allocated ID 2 I190903 20:12:51.316745 10 gossip/gossip.go:394 [n2] NodeDescriptor set to node_id:2 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567541571316691545 cluster_name:"" sql_address: I190903 20:12:51.316803 10 storage/stores.go:240 [n2] read 0 node addresses from persistent storage I190903 20:12:51.316910 10 storage/stores.go:259 [n2] wrote 1 node addresses to persistent storage I190903 20:12:51.320949 59 storage/stores.go:259 [n1] wrote 1 node addresses to persistent storage I190903 20:12:51.323059 10 server/node.go:611 [n2] bootstrapped store [n2,s2] I190903 20:12:51.323192 10 server/node.go:503 [n2] node=2: started with [n2=] engine(s) and attributes [] I190903 20:12:51.323257 10 server/server.go:1989 [n2] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 20:12:51.323323 10 server/server.go:1486 [n2] starting https server at 127.0.0.1:35161 (use: 127.0.0.1:35161) I190903 20:12:51.323348 10 server/server.go:1491 [n2] starting postgres server at 127.0.0.1:46337 (use: 127.0.0.1:46337) I190903 20:12:51.323366 10 server/server.go:1493 [n2] starting grpc server at 127.0.0.1:33569 I190903 20:12:51.323382 10 server/server.go:1494 [n2] advertising CockroachDB node at 127.0.0.1:33569 W190903 20:12:51.327603 10 jobs/registry.go:339 [n2] unable to get node liveness: node not in the liveness table I190903 20:12:51.328207 10 server/server.go:1562 [n2] done ensuring all necessary migrations have run I190903 20:12:51.328295 10 server/server.go:1800 [n2] serving sql connections I190903 20:12:51.339607 1826 sql/event_log.go:130 [n2] Event: "node_join", target: 2, info: {Descriptor:{NodeID:2 Address:127.0.0.1:33569 Attrs: Locality:region=test,dc=dc2 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567541571316691545 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:46337} ClusterID:b736962c-f08c-476e-9b06-40a804863214 StartedAt:1567541571316691545 LastUp:1567541571316691545} I190903 20:12:51.350244 1504 server/server_update.go:68 [n2] no need to upgrade, cluster already at the newest version I190903 20:12:51.373572 1866 sql/event_log.go:130 [n2,client=127.0.0.1:36440,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 20:12:51.376140 10 base/addr_validation.go:296 [n?] server certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node I190903 20:12:51.376164 10 base/addr_validation.go:342 [n?] web UI certificate addresses: IP=127.0.0.1,::1; DNS=localhost,*.local; CN=node W190903 20:12:51.385229 10 server/status/runtime.go:308 [n?] Could not parse build timestamp: parsing time "" as "2006/01/02 15:04:05": cannot parse "" as "2006" I190903 20:12:51.392427 10 server/server.go:911 [n?] monitoring forward clock jumps based on server.clock.forward_jump_check_enabled I190903 20:12:51.403226 10 server/config.go:502 [n?] 1 storage engine initialized I190903 20:12:51.403252 10 server/config.go:505 [n?] RocksDB cache size: 128 MiB I190903 20:12:51.403262 10 server/config.go:505 [n?] store 0: in-memory, size 0 B W190903 20:12:51.403346 10 gossip/gossip.go:1517 [n?] no incoming or outgoing connections I190903 20:12:51.403397 10 server/server.go:1359 [n?] no stores bootstrapped and --join flag specified, awaiting init command or join with an already initialized node. I190903 20:12:51.409544 1897 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.412148 1897 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.413475 1897 storage/store_snapshot.go:995 [n1,replicate,s1,r4/1:/System{/tsd-tse}] sending LEARNER snapshot c8f7b291 at applied index 23 I190903 20:12:51.418223 1897 storage/store_snapshot.go:1038 [n1,replicate,s1,r4/1:/System{/tsd-tse}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 1301, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:51.418892 2057 gossip/client.go:124 [n?] started gossip client to 127.0.0.1:43339 I190903 20:12:51.419114 1885 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:35063 I190903 20:12:51.419782 10 server/node.go:636 [n?] connecting to gossip network to verify cluster ID... I190903 20:12:51.419810 10 server/node.go:656 [n?] node connected via gossip and verified as part of cluster "b736962c-f08c-476e-9b06-40a804863214" I190903 20:12:51.420789 1885 gossip/server.go:227 [n1] received initial cluster-verification connection from 127.0.0.1:35063 I190903 20:12:51.421411 1903 storage/replica_raftstorage.go:793 [n2,s2,r4/2:{-}] applying LEARNER snapshot [id=c8f7b291 index=23] I190903 20:12:51.422357 1903 storage/replica_raftstorage.go:814 [n2,s2,r4/2:/System{/tsd-tse}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=c8f7b291 index=23] I190903 20:12:51.423150 1897 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [(n2,s2):2] remove []): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.425732 1897 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.426270 1897 storage/replica_command.go:1430 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.434346 1897 storage/replica_raft.go:291 [n1,s1,r4/1:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:51.434392 1897 storage/replica_raft.go:304 [n1,s1,r4/1:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:51.435333 1897 storage/replica_command.go:979 [n1,replicate,s1,r4/1:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n1,s1,r4/1:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:51.435383 1897 storage/queue.go:1027 [n1,replicate,s1,r4/1:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n1,s1,r4/1:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:51.447920 10 server/node.go:372 [n?] new node allocated ID 3 I190903 20:12:51.448010 10 gossip/gossip.go:394 [n3] NodeDescriptor set to node_id:3 address: attrs:<> locality: tiers: > ServerVersion: build_tag:"v19.2.0-alpha.20190606-1964-g982579c" started_at:1567541571447946077 cluster_name:"" sql_address: I190903 20:12:51.448091 10 storage/stores.go:240 [n3] read 0 node addresses from persistent storage I190903 20:12:51.448131 10 storage/stores.go:259 [n3] wrote 2 node addresses to persistent storage I190903 20:12:51.448793 59 storage/stores.go:259 [n1] wrote 2 node addresses to persistent storage I190903 20:12:51.449162 1331 storage/stores.go:259 [n2] wrote 2 node addresses to persistent storage I190903 20:12:51.449691 10 server/node.go:611 [n3] bootstrapped store [n3,s3] I190903 20:12:51.449798 10 server/node.go:503 [n3] node=3: started with [n3=] engine(s) and attributes [] I190903 20:12:51.449867 10 server/server.go:1989 [n3] Could not start goroutine dumper worker due to: directory to store dumps could not be determined I190903 20:12:51.449931 10 server/server.go:1486 [n3] starting https server at 127.0.0.1:39607 (use: 127.0.0.1:39607) I190903 20:12:51.449970 10 server/server.go:1491 [n3] starting postgres server at 127.0.0.1:46523 (use: 127.0.0.1:46523) I190903 20:12:51.449987 10 server/server.go:1493 [n3] starting grpc server at 127.0.0.1:35063 I190903 20:12:51.450952 10 server/server.go:1494 [n3] advertising CockroachDB node at 127.0.0.1:35063 I190903 20:12:51.452891 499 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.455267 499 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.455590 10 server/server.go:1562 [n3] done ensuring all necessary migrations have run I190903 20:12:51.455644 10 server/server.go:1800 [n3] serving sql connections I190903 20:12:51.458090 499 storage/store_snapshot.go:995 [n1,replicate,s1,r9/1:/Table/1{3-4}] sending LEARNER snapshot a8881868 at applied index 23 I190903 20:12:51.458242 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r9/1:/Table/1{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 21, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.461758 2408 storage/replica_raftstorage.go:793 [n2,s2,r9/2:{-}] applying LEARNER snapshot [id=a8881868 index=23] I190903 20:12:51.463300 2408 storage/replica_raftstorage.go:814 [n2,s2,r9/2:/Table/1{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a8881868 index=23] I190903 20:12:51.463782 499 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.467584 499 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.468776 499 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.477675 499 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.480517 499 storage/store_snapshot.go:995 [n1,replicate,s1,r9/1:/Table/1{3-4}] sending LEARNER snapshot bc7b8066 at applied index 29 I190903 20:12:51.480684 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r9/1:/Table/1{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 35, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.481342 2369 sql/event_log.go:130 [n3] Event: "node_join", target: 3, info: {Descriptor:{NodeID:3 Address:127.0.0.1:35063 Attrs: Locality:region=test,dc=dc3 ServerVersion:19.1-9 BuildTag:v19.2.0-alpha.20190606-1964-g982579c StartedAt:1567541571447946077 LocalityAddress:[] ClusterName: SQLAddress:127.0.0.1:46523} ClusterID:b736962c-f08c-476e-9b06-40a804863214 StartedAt:1567541571447946077 LastUp:1567541571447946077} I190903 20:12:51.483522 2367 server/server_update.go:68 [n3] no need to upgrade, cluster already at the newest version I190903 20:12:51.484195 2474 storage/replica_raftstorage.go:793 [n3,s3,r9/3:{-}] applying LEARNER snapshot [id=bc7b8066 index=29] I190903 20:12:51.485623 2474 storage/replica_raftstorage.go:814 [n3,s3,r9/3:/Table/1{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=bc7b8066 index=29] I190903 20:12:51.486376 499 storage/replica_command.go:1430 [n1,replicate,s1,r9/1:/Table/1{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r9:/Table/1{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.490485 499 storage/replica_raft.go:291 [n1,s1,r9/1:/Table/1{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.492484 499 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.495682 499 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.496605 499 storage/store_snapshot.go:995 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] sending LEARNER snapshot 1e835e5b at applied index 17 I190903 20:12:51.496743 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.497838 2382 sql/event_log.go:130 [n3,client=127.0.0.1:57334,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:kv.range_split.by_load_enabled Value:false User:root} I190903 20:12:51.498706 2495 storage/replica_raftstorage.go:793 [n2,s2,r5/2:{-}] applying LEARNER snapshot [id=1e835e5b index=17] I190903 20:12:51.500139 2495 storage/replica_raftstorage.go:814 [n2,s2,r5/2:/{Systemtse-Table/System…}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=1e835e5b index=17] I190903 20:12:51.500625 499 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n2,s2):2] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.506817 499 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.508015 499 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.519745 499 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.521194 499 storage/store_snapshot.go:995 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] sending LEARNER snapshot 210f8646 at applied index 21 I190903 20:12:51.521322 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.522489 2629 storage/replica_raftstorage.go:793 [n3,s3,r5/3:{-}] applying LEARNER snapshot [id=210f8646 index=21] I190903 20:12:51.523610 2629 storage/replica_raftstorage.go:814 [n3,s3,r5/3:/{Systemtse-Table/System…}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=210f8646 index=21] I190903 20:12:51.524071 499 storage/replica_command.go:1430 [n1,replicate,s1,r5/1:/{Systemtse-Table/System…}] change replicas (add [(n3,s3):3] remove []): existing descriptor r5:/{Systemtse-Table/SystemConfigSpan/Start} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.532548 499 storage/replica_raft.go:291 [n1,s1,r5/1:/{Systemtse-Table/System…}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.533974 499 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.536320 499 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:51.538044 499 storage/store_snapshot.go:995 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] sending LEARNER snapshot 6157dea9 at applied index 48 I190903 20:12:51.538501 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 38, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.542868 2126 storage/replica_raftstorage.go:793 [n3,s3,r3/2:{-}] applying LEARNER snapshot [id=6157dea9 index=48] I190903 20:12:51.544428 2126 storage/replica_raftstorage.go:814 [n3,s3,r3/2:/System/{NodeLive…-tsd}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=6157dea9 index=48] I190903 20:12:51.545173 499 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n3,s3):2] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 20:12:51.552600 499 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:51.553352 499 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 20:12:51.562066 499 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:51.564047 499 storage/store_snapshot.go:995 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] sending LEARNER snapshot cc8919f1 at applied index 52 I190903 20:12:51.564425 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 42, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.566258 2751 storage/replica_raftstorage.go:793 [n2,s2,r3/3:{-}] applying LEARNER snapshot [id=cc8919f1 index=52] I190903 20:12:51.567507 2751 storage/replica_raftstorage.go:814 [n2,s2,r3/3:/System/{NodeLive…-tsd}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=cc8919f1 index=52] I190903 20:12:51.567931 499 storage/replica_command.go:1430 [n1,replicate,s1,r3/1:/System/{NodeLive…-tsd}] change replicas (add [(n2,s2):3] remove []): existing descriptor r3:/System/{NodeLivenessMax-tsd} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 20:12:51.570262 499 storage/replica_raft.go:291 [n1,s1,r3/1:/System/{NodeLive…-tsd}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 20:12:51.572517 499 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.575531 499 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.577052 499 storage/store_snapshot.go:995 [n1,replicate,s1,r7/1:/Table/1{1-2}] sending LEARNER snapshot e02c99e2 at applied index 21 I190903 20:12:51.577184 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r7/1:/Table/1{1-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.578149 2711 storage/replica_raftstorage.go:793 [n2,s2,r7/2:{-}] applying LEARNER snapshot [id=e02c99e2 index=21] I190903 20:12:51.579396 2711 storage/replica_raftstorage.go:814 [n2,s2,r7/2:/Table/1{1-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e02c99e2 index=21] I190903 20:12:51.579945 499 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.587746 499 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.588243 499 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.596740 499 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.598284 499 storage/store_snapshot.go:995 [n1,replicate,s1,r7/1:/Table/1{1-2}] sending LEARNER snapshot ef266cf9 at applied index 25 I190903 20:12:51.598406 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r7/1:/Table/1{1-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.599190 2836 storage/replica_raftstorage.go:793 [n3,s3,r7/3:{-}] applying LEARNER snapshot [id=ef266cf9 index=25] I190903 20:12:51.600151 2836 storage/replica_raftstorage.go:814 [n3,s3,r7/3:/Table/1{1-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ef266cf9 index=25] I190903 20:12:51.600650 499 storage/replica_command.go:1430 [n1,replicate,s1,r7/1:/Table/1{1-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r7:/Table/1{1-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.604858 499 storage/replica_raft.go:291 [n1,s1,r7/1:/Table/1{1-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.607088 499 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.609889 499 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:51.610780 499 storage/store_snapshot.go:995 [n1,replicate,s1,r10/1:/Table/1{4-5}] sending LEARNER snapshot fa908693 at applied index 16 I190903 20:12:51.610952 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r10/1:/Table/1{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.612044 2857 storage/replica_raftstorage.go:793 [n3,s3,r10/2:{-}] applying LEARNER snapshot [id=fa908693 index=16] I190903 20:12:51.621334 2857 storage/replica_raftstorage.go:814 [n3,s3,r10/2:/Table/1{4-5}] applied LEARNER snapshot [total=4ms ingestion=4@4ms id=fa908693 index=16] I190903 20:12:51.621950 499 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 20:12:51.628579 499 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:51.629125 499 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 20:12:51.637749 499 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:51.639244 499 storage/store_snapshot.go:995 [n1,replicate,s1,r10/1:/Table/1{4-5}] sending LEARNER snapshot 752f73b2 at applied index 20 I190903 20:12:51.639370 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r10/1:/Table/1{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.640906 2956 storage/replica_raftstorage.go:793 [n2,s2,r10/3:{-}] applying LEARNER snapshot [id=752f73b2 index=20] I190903 20:12:51.642573 2956 storage/replica_raftstorage.go:814 [n2,s2,r10/3:/Table/1{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=752f73b2 index=20] I190903 20:12:51.643358 499 storage/replica_command.go:1430 [n1,replicate,s1,r10/1:/Table/1{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r10:/Table/1{4-5} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 20:12:51.649021 499 storage/replica_raft.go:291 [n1,s1,r10/1:/Table/1{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 20:12:51.650496 499 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.653078 499 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.654424 499 storage/store_snapshot.go:995 [n1,replicate,s1,r13/1:/Table/1{7-8}] sending LEARNER snapshot 280a84f2 at applied index 16 I190903 20:12:51.654528 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r13/1:/Table/1{7-8}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.655607 2928 storage/replica_raftstorage.go:793 [n2,s2,r13/2:{-}] applying LEARNER snapshot [id=280a84f2 index=16] I190903 20:12:51.656690 2928 storage/replica_raftstorage.go:814 [n2,s2,r13/2:/Table/1{7-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=280a84f2 index=16] I190903 20:12:51.658014 499 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n2,s2):2] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.661201 499 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.661699 499 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.669744 499 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.673467 499 storage/store_snapshot.go:995 [n1,replicate,s1,r13/1:/Table/1{7-8}] sending LEARNER snapshot 8b51e6cf at applied index 20 I190903 20:12:51.674186 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r13/1:/Table/1{7-8}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.676083 2943 storage/replica_raftstorage.go:793 [n3,s3,r13/3:{-}] applying LEARNER snapshot [id=8b51e6cf index=20] I190903 20:12:51.679304 2943 storage/replica_raftstorage.go:814 [n3,s3,r13/3:/Table/1{7-8}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=8b51e6cf index=20] I190903 20:12:51.680019 499 storage/replica_command.go:1430 [n1,replicate,s1,r13/1:/Table/1{7-8}] change replicas (add [(n3,s3):3] remove []): existing descriptor r13:/Table/1{7-8} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.682688 499 storage/replica_raft.go:291 [n1,s1,r13/1:/Table/1{7-8}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.684739 499 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.688548 499 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.689506 499 storage/store_snapshot.go:995 [n1,replicate,s1,r15/1:/Table/{19-20}] sending LEARNER snapshot e3dd4fa4 at applied index 16 I190903 20:12:51.689614 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r15/1:/Table/{19-20}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.690741 3114 storage/replica_raftstorage.go:793 [n2,s2,r15/2:{-}] applying LEARNER snapshot [id=e3dd4fa4 index=16] I190903 20:12:51.692218 3114 storage/replica_raftstorage.go:814 [n2,s2,r15/2:/Table/{19-20}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e3dd4fa4 index=16] I190903 20:12:51.692662 499 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n2,s2):2] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.700233 499 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.701099 499 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.709293 499 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.712482 499 storage/store_snapshot.go:995 [n1,replicate,s1,r15/1:/Table/{19-20}] sending LEARNER snapshot f70ba18e at applied index 20 I190903 20:12:51.712607 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r15/1:/Table/{19-20}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.713772 3145 storage/replica_raftstorage.go:793 [n3,s3,r15/3:{-}] applying LEARNER snapshot [id=f70ba18e index=20] I190903 20:12:51.715347 3145 storage/replica_raftstorage.go:814 [n3,s3,r15/3:/Table/{19-20}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=f70ba18e index=20] I190903 20:12:51.716952 499 storage/replica_command.go:1430 [n1,replicate,s1,r15/1:/Table/{19-20}] change replicas (add [(n3,s3):3] remove []): existing descriptor r15:/Table/{19-20} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.719940 499 storage/replica_raft.go:291 [n1,s1,r15/1:/Table/{19-20}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.721096 499 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.724295 499 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.725740 499 storage/store_snapshot.go:995 [n1,replicate,s1,r19/1:/Table/2{3-4}] sending LEARNER snapshot fe50d376 at applied index 17 I190903 20:12:51.725872 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r19/1:/Table/2{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.727608 3191 storage/replica_raftstorage.go:793 [n2,s2,r19/2:{-}] applying LEARNER snapshot [id=fe50d376 index=17] I190903 20:12:51.728974 3191 storage/replica_raftstorage.go:814 [n2,s2,r19/2:/Table/2{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=fe50d376 index=17] I190903 20:12:51.729530 499 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.734220 499 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.735195 499 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.746453 499 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.748347 499 storage/store_snapshot.go:995 [n1,replicate,s1,r19/1:/Table/2{3-4}] sending LEARNER snapshot e5c075ef at applied index 21 I190903 20:12:51.748838 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r19/1:/Table/2{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.750696 3284 storage/replica_raftstorage.go:793 [n3,s3,r19/3:{-}] applying LEARNER snapshot [id=e5c075ef index=21] I190903 20:12:51.751967 3284 storage/replica_raftstorage.go:814 [n3,s3,r19/3:/Table/2{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e5c075ef index=21] I190903 20:12:51.752455 499 storage/replica_command.go:1430 [n1,replicate,s1,r19/1:/Table/2{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r19:/Table/2{3-4} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.766571 499 storage/replica_raft.go:291 [n1,s1,r19/1:/Table/2{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.767992 499 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.771516 499 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:51.772482 499 storage/store_snapshot.go:995 [n1,replicate,s1,r18/1:/Table/2{2-3}] sending LEARNER snapshot 45559b0b at applied index 16 I190903 20:12:51.772606 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r18/1:/Table/2{2-3}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.773881 1120 storage/replica_raftstorage.go:793 [n3,s3,r18/2:{-}] applying LEARNER snapshot [id=45559b0b index=16] I190903 20:12:51.775349 1120 storage/replica_raftstorage.go:814 [n3,s3,r18/2:/Table/2{2-3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=45559b0b index=16] I190903 20:12:51.775939 499 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n3,s3):2] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 20:12:51.786692 499 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:51.787300 499 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 20:12:51.795444 499 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:51.797275 499 storage/store_snapshot.go:995 [n1,replicate,s1,r18/1:/Table/2{2-3}] sending LEARNER snapshot d4a5bd9f at applied index 20 I190903 20:12:51.797424 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r18/1:/Table/2{2-3}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.798686 3438 storage/replica_raftstorage.go:793 [n2,s2,r18/3:{-}] applying LEARNER snapshot [id=d4a5bd9f index=20] I190903 20:12:51.801097 3438 storage/replica_raftstorage.go:814 [n2,s2,r18/3:/Table/2{2-3}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=d4a5bd9f index=20] I190903 20:12:51.802071 499 storage/replica_command.go:1430 [n1,replicate,s1,r18/1:/Table/2{2-3}] change replicas (add [(n2,s2):3] remove []): existing descriptor r18:/Table/2{2-3} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 20:12:51.805815 499 storage/replica_raft.go:291 [n1,s1,r18/1:/Table/2{2-3}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 20:12:51.810058 499 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.812496 240 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:12:51.812883 499 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.813813 499 storage/store_snapshot.go:995 [n1,replicate,s1,r11/1:/Table/1{5-6}] sending LEARNER snapshot e720d1b5 at applied index 16 I190903 20:12:51.813913 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r11/1:/Table/1{5-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.815104 3355 storage/replica_raftstorage.go:793 [n2,s2,r11/2:{-}] applying LEARNER snapshot [id=e720d1b5 index=16] I190903 20:12:51.816634 3355 storage/replica_raftstorage.go:814 [n2,s2,r11/2:/Table/1{5-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e720d1b5 index=16] I190903 20:12:51.817106 499 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.821305 499 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.822450 499 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.830408 499 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.831695 499 storage/store_snapshot.go:995 [n1,replicate,s1,r11/1:/Table/1{5-6}] sending LEARNER snapshot a8db7616 at applied index 20 I190903 20:12:51.831827 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r11/1:/Table/1{5-6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.835380 3562 storage/replica_raftstorage.go:793 [n3,s3,r11/3:{-}] applying LEARNER snapshot [id=a8db7616 index=20] I190903 20:12:51.836960 3562 storage/replica_raftstorage.go:814 [n3,s3,r11/3:/Table/1{5-6}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=a8db7616 index=20] I190903 20:12:51.838094 499 storage/replica_command.go:1430 [n1,replicate,s1,r11/1:/Table/1{5-6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r11:/Table/1{5-6} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.843443 499 storage/replica_raft.go:291 [n1,s1,r11/1:/Table/1{5-6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.844549 499 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.847054 499 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.848062 499 storage/store_snapshot.go:995 [n1,replicate,s1,r8/1:/Table/1{2-3}] sending LEARNER snapshot 49e38368 at applied index 44 I190903 20:12:51.848322 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r8/1:/Table/1{2-3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 66, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.850462 3278 storage/replica_raftstorage.go:793 [n2,s2,r8/2:{-}] applying LEARNER snapshot [id=49e38368 index=44] I190903 20:12:51.852065 3278 storage/replica_raftstorage.go:814 [n2,s2,r8/2:/Table/1{2-3}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=49e38368 index=44] I190903 20:12:51.852839 499 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.855881 499 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.856482 499 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.865563 499 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.867506 499 storage/store_snapshot.go:995 [n1,replicate,s1,r8/1:/Table/1{2-3}] sending LEARNER snapshot 8a7aedbd at applied index 48 I190903 20:12:51.867853 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r8/1:/Table/1{2-3}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 70, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.870469 3658 storage/replica_raftstorage.go:793 [n3,s3,r8/3:{-}] applying LEARNER snapshot [id=8a7aedbd index=48] I190903 20:12:51.874662 3658 storage/replica_raftstorage.go:814 [n3,s3,r8/3:/Table/1{2-3}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=8a7aedbd index=48] I190903 20:12:51.875465 499 storage/replica_command.go:1430 [n1,replicate,s1,r8/1:/Table/1{2-3}] change replicas (add [(n3,s3):3] remove []): existing descriptor r8:/Table/1{2-3} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.878873 499 storage/replica_raft.go:291 [n1,s1,r8/1:/Table/1{2-3}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.880352 499 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.885138 499 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.886156 499 storage/store_snapshot.go:995 [n1,replicate,s1,r14/1:/Table/1{8-9}] sending LEARNER snapshot 6ba7f1fb at applied index 16 I190903 20:12:51.886261 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r14/1:/Table/1{8-9}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.887837 3678 storage/replica_raftstorage.go:793 [n2,s2,r14/2:{-}] applying LEARNER snapshot [id=6ba7f1fb index=16] I190903 20:12:51.889166 3678 storage/replica_raftstorage.go:814 [n2,s2,r14/2:/Table/1{8-9}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=6ba7f1fb index=16] I190903 20:12:51.890533 499 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n2,s2):2] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.895237 499 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.895634 499 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.911360 499 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.914632 499 storage/store_snapshot.go:995 [n1,replicate,s1,r14/1:/Table/1{8-9}] sending LEARNER snapshot 28ee3c74 at applied index 20 I190903 20:12:51.914995 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r14/1:/Table/1{8-9}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.916653 3860 storage/replica_raftstorage.go:793 [n3,s3,r14/3:{-}] applying LEARNER snapshot [id=28ee3c74 index=20] I190903 20:12:51.918324 3860 storage/replica_raftstorage.go:814 [n3,s3,r14/3:/Table/1{8-9}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=28ee3c74 index=20] I190903 20:12:51.919299 499 storage/replica_command.go:1430 [n1,replicate,s1,r14/1:/Table/1{8-9}] change replicas (add [(n3,s3):3] remove []): existing descriptor r14:/Table/1{8-9} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.940005 499 storage/replica_raft.go:291 [n1,s1,r14/1:/Table/1{8-9}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.944010 499 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, next=2, gen=0] I190903 20:12:51.948852 499 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:51.951780 499 storage/store_snapshot.go:995 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] sending LEARNER snapshot 8efce5a0 at applied index 112 I190903 20:12:51.952030 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 79, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.954556 3681 storage/replica_raftstorage.go:793 [n2,s2,r1/2:{-}] applying LEARNER snapshot [id=8efce5a0 index=112] I190903 20:12:51.956130 3681 storage/replica_raftstorage.go:814 [n2,s2,r1/2:/{Min-System/NodeL…}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=8efce5a0 index=112] I190903 20:12:51.958562 499 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n2,s2):2] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:51.963863 499 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:51.964411 499 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:51.970239 499 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:51.972176 499 storage/store_snapshot.go:995 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] sending LEARNER snapshot 5ce218c8 at applied index 118 I190903 20:12:51.972452 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 87, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:51.973839 3751 storage/replica_raftstorage.go:793 [n3,s3,r1/3:{-}] applying LEARNER snapshot [id=5ce218c8 index=118] I190903 20:12:51.975195 3751 storage/replica_raftstorage.go:814 [n3,s3,r1/3:/{Min-System/NodeL…}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5ce218c8 index=118] I190903 20:12:51.975816 499 storage/replica_command.go:1430 [n1,replicate,s1,r1/1:/{Min-System/NodeL…}] change replicas (add [(n3,s3):3] remove []): existing descriptor r1:/{Min-System/NodeLiveness} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:51.996835 499 storage/replica_raft.go:291 [n1,s1,r1/1:/{Min-System/NodeL…}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:51.998128 499 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.003229 499 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:52.006665 499 storage/store_snapshot.go:995 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] sending LEARNER snapshot 660d182f at applied index 44 I190903 20:12:52.008223 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 63, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.011675 4059 storage/replica_raftstorage.go:793 [n2,s2,r6/2:{-}] applying LEARNER snapshot [id=660d182f index=44] I190903 20:12:52.019797 4059 storage/replica_raftstorage.go:814 [n2,s2,r6/2:/Table/{SystemCon…-11}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=660d182f index=44] I190903 20:12:52.020338 499 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n2,s2):2] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:52.024000 499 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:52.024610 499 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:52.034763 499 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:52.036093 499 storage/store_snapshot.go:995 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] sending LEARNER snapshot a24abf16 at applied index 48 I190903 20:12:52.036372 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 67, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.037992 3776 storage/replica_raftstorage.go:793 [n3,s3,r6/3:{-}] applying LEARNER snapshot [id=a24abf16 index=48] I190903 20:12:52.039454 3776 storage/replica_raftstorage.go:814 [n3,s3,r6/3:/Table/{SystemCon…-11}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a24abf16 index=48] I190903 20:12:52.040068 499 storage/replica_command.go:1430 [n1,replicate,s1,r6/1:/Table/{SystemCon…-11}] change replicas (add [(n3,s3):3] remove []): existing descriptor r6:/Table/{SystemConfigSpan/Start-11} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:52.044089 499 storage/replica_raft.go:291 [n1,s1,r6/1:/Table/{SystemCon…-11}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:52.045183 499 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.048198 499 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:52.049332 499 storage/store_snapshot.go:995 [n1,replicate,s1,r12/1:/Table/1{6-7}] sending LEARNER snapshot 889e8a2f at applied index 16 I190903 20:12:52.049448 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r12/1:/Table/1{6-7}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.054265 4166 storage/replica_raftstorage.go:793 [n3,s3,r12/2:{-}] applying LEARNER snapshot [id=889e8a2f index=16] I190903 20:12:52.057382 4166 storage/replica_raftstorage.go:814 [n3,s3,r12/2:/Table/1{6-7}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=889e8a2f index=16] I190903 20:12:52.059148 499 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n3,s3):2] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 20:12:52.062614 499 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:52.063135 499 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 20:12:52.074481 499 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:52.075937 499 storage/store_snapshot.go:995 [n1,replicate,s1,r12/1:/Table/1{6-7}] sending LEARNER snapshot b5a5aa3b at applied index 20 I190903 20:12:52.076166 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r12/1:/Table/1{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.077374 4316 storage/replica_raftstorage.go:793 [n2,s2,r12/3:{-}] applying LEARNER snapshot [id=b5a5aa3b index=20] I190903 20:12:52.081045 4316 storage/replica_raftstorage.go:814 [n2,s2,r12/3:/Table/1{6-7}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=b5a5aa3b index=20] I190903 20:12:52.082613 499 storage/replica_command.go:1430 [n1,replicate,s1,r12/1:/Table/1{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r12:/Table/1{6-7} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 20:12:52.086349 499 storage/replica_raft.go:291 [n1,s1,r12/1:/Table/1{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 20:12:52.087555 499 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.091250 499 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:52.102445 499 storage/store_snapshot.go:995 [n1,replicate,s1,r20/1:/{Table/24-Max}] sending LEARNER snapshot 7dbd5b1b at applied index 16 I190903 20:12:52.103051 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r20/1:/{Table/24-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:52.104368 4414 storage/replica_raftstorage.go:793 [n2,s2,r20/2:{-}] applying LEARNER snapshot [id=7dbd5b1b index=16] I190903 20:12:52.110680 4414 storage/replica_raftstorage.go:814 [n2,s2,r20/2:/{Table/24-Max}] applied LEARNER snapshot [total=6ms ingestion=4@6ms id=7dbd5b1b index=16] I190903 20:12:52.111731 499 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:52.115628 499 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:52.116150 499 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:52.129152 499 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:52.130966 499 storage/store_snapshot.go:995 [n1,replicate,s1,r20/1:/{Table/24-Max}] sending LEARNER snapshot 2868e498 at applied index 20 I190903 20:12:52.131141 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r20/1:/{Table/24-Max}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.132314 4563 storage/replica_raftstorage.go:793 [n3,s3,r20/3:{-}] applying LEARNER snapshot [id=2868e498 index=20] I190903 20:12:52.134055 4563 storage/replica_raftstorage.go:814 [n3,s3,r20/3:/{Table/24-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=2868e498 index=20] I190903 20:12:52.134957 499 storage/replica_command.go:1430 [n1,replicate,s1,r20/1:/{Table/24-Max}] change replicas (add [(n3,s3):3] remove []): existing descriptor r20:/{Table/24-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:52.139586 499 storage/replica_raft.go:291 [n1,s1,r20/1:/{Table/24-Max}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:52.141211 499 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.144286 499 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:52.145103 499 storage/store_snapshot.go:995 [n1,replicate,s1,r17/1:/Table/2{1-2}] sending LEARNER snapshot 43da385d at applied index 24 I190903 20:12:52.145229 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r17/1:/Table/2{1-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.146326 4613 storage/replica_raftstorage.go:793 [n2,s2,r17/2:{-}] applying LEARNER snapshot [id=43da385d index=24] I190903 20:12:52.147985 4613 storage/replica_raftstorage.go:814 [n2,s2,r17/2:/Table/2{1-2}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=43da385d index=24] I190903 20:12:52.149405 499 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:52.153443 499 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:52.154027 499 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:52.166283 499 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:52.171641 499 storage/store_snapshot.go:995 [n1,replicate,s1,r17/1:/Table/2{1-2}] sending LEARNER snapshot 931fa4bb at applied index 28 I190903 20:12:52.171813 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r17/1:/Table/2{1-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.173094 4674 storage/replica_raftstorage.go:793 [n3,s3,r17/3:{-}] applying LEARNER snapshot [id=931fa4bb index=28] I190903 20:12:52.175660 4674 storage/replica_raftstorage.go:814 [n3,s3,r17/3:/Table/2{1-2}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=931fa4bb index=28] I190903 20:12:52.176082 499 storage/replica_command.go:1430 [n1,replicate,s1,r17/1:/Table/2{1-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r17:/Table/2{1-2} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:52.179962 499 storage/replica_raft.go:291 [n1,s1,r17/1:/Table/2{1-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:52.182773 499 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.192384 499 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:52.195301 499 storage/store_snapshot.go:995 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] sending LEARNER snapshot 3d80f60b at applied index 20 I190903 20:12:52.195429 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.196509 4807 storage/replica_raftstorage.go:793 [n2,s2,r2/2:{-}] applying LEARNER snapshot [id=3d80f60b index=20] I190903 20:12:52.198061 4807 storage/replica_raftstorage.go:814 [n2,s2,r2/2:/System/NodeLiveness{-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=3d80f60b index=20] I190903 20:12:52.198459 499 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=1] I190903 20:12:52.201895 499 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:52.202809 499 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:52.211464 499 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:52.214231 499 storage/store_snapshot.go:995 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] sending LEARNER snapshot 87228902 at applied index 24 I190903 20:12:52.214363 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.215467 4657 storage/replica_raftstorage.go:793 [n3,s3,r2/3:{-}] applying LEARNER snapshot [id=87228902 index=24] I190903 20:12:52.219265 4657 storage/replica_raftstorage.go:814 [n3,s3,r2/3:/System/NodeLiveness{-Max}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=87228902 index=24] I190903 20:12:52.220803 499 storage/replica_command.go:1430 [n1,replicate,s1,r2/1:/System/NodeLiveness{-Max}] change replicas (add [(n3,s3):3] remove []): existing descriptor r2:/System/NodeLiveness{-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3LEARNER, next=4, gen=3] I190903 20:12:52.224348 499 storage/replica_raft.go:291 [n1,s1,r2/1:/System/NodeLiveness{-Max}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2 (n3,s3):3] next=4 I190903 20:12:52.226062 499 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, next=2, gen=0] I190903 20:12:52.228909 499 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:52.230664 499 storage/store_snapshot.go:995 [n1,replicate,s1,r16/1:/Table/2{0-1}] sending LEARNER snapshot eb7241c3 at applied index 16 I190903 20:12:52.230841 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r16/1:/Table/2{0-1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.232305 4767 storage/replica_raftstorage.go:793 [n3,s3,r16/2:{-}] applying LEARNER snapshot [id=eb7241c3 index=16] I190903 20:12:52.234328 4767 storage/replica_raftstorage.go:814 [n3,s3,r16/2:/Table/2{0-1}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=eb7241c3 index=16] I190903 20:12:52.237318 499 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=1] I190903 20:12:52.240785 499 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:52.241720 499 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, next=3, gen=2] I190903 20:12:52.250153 499 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:52.253023 499 storage/store_snapshot.go:995 [n1,replicate,s1,r16/1:/Table/2{0-1}] sending LEARNER snapshot 38ae6af2 at applied index 20 I190903 20:12:52.253148 499 storage/store_snapshot.go:1038 [n1,replicate,s1,r16/1:/Table/2{0-1}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.254621 4901 storage/replica_raftstorage.go:793 [n2,s2,r16/3:{-}] applying LEARNER snapshot [id=38ae6af2 index=20] I190903 20:12:52.259922 4901 storage/replica_raftstorage.go:814 [n2,s2,r16/3:/Table/2{0-1}] applied LEARNER snapshot [total=5ms ingestion=4@1ms id=38ae6af2 index=20] I190903 20:12:52.264369 499 storage/replica_command.go:1430 [n1,replicate,s1,r16/1:/Table/2{0-1}] change replicas (add [(n2,s2):3] remove []): existing descriptor r16:/Table/2{0-1} [(n1,s1):1, (n3,s3):2, (n2,s2):3LEARNER, next=4, gen=3] I190903 20:12:52.267715 499 storage/replica_raft.go:291 [n1,s1,r16/1:/Table/2{0-1}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):1 (n3,s3):2 (n2,s2):3] next=4 I190903 20:12:52.268985 499 storage/queue.go:1127 [n1,replicate] purgatory is now empty I190903 20:12:52.281496 10 testutils/testcluster/testcluster.go:669 WaitForFullReplication took: 783.237949ms I190903 20:12:52.297820 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 52, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:52.306375 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:server.declined_reservation_timeout Value:00:00:00 User:root} I190903 20:12:52.312558 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_cluster_setting", target: 0, info: {SettingName:server.failed_reservation_timeout Value:00:00:00 User:root} === RUN TestRepartitioning/unpartitioned/unpartitioned I190903 20:12:52.322891 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 52, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[]} I190903 20:12:52.347047 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 53, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:52.374018 4904 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r4:/System{/tsd-tse} [(n1,s1):1, (n2,s2):2, next=3, gen=2] I190903 20:12:52.375610 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 54, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 20:12:52.378162 3745 storage/replica_command.go:284 [n1,split,s1,r20/1:/{Table/24-Max}] initiating a split of this range at key /Table/54 [r21] (zone config) === RUN TestRepartitioning/unpartitioned/single_col_list_partitioning I190903 20:12:52.398837 4904 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:52.401165 4904 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, next=3, gen=3] I190903 20:12:52.411176 5557 storage/replica_consistency.go:192 [n2,consistencyChecker,s2,r4/2:/System{/tsd-tse}] triggering stats recomputation to resolve delta of {ContainsEstimates:true LastUpdateNanos:1567541571455507996 IntentAge:0 GCBytesAge:0 LiveBytes:-97968 LiveCount:-1941 KeyBytes:-93930 KeyCount:-1941 ValBytes:-4038 ValCount:-1941 IntentBytes:0 IntentCount:0 SysBytes:0 SysCount:0} I190903 20:12:52.420867 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=54 (unpartitioned) version=2 mtime=2019-09-03 20:12:52.399078964 +0000 UTC I190903 20:12:52.421560 4904 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:12:52.424637 4904 storage/store_snapshot.go:995 [n2,replicate,s2,r4/2:/System{/tsd-tse}] sending LEARNER snapshot 40220d07 at applied index 42 W190903 20:12:52.425248 6291 storage/intentresolver/intent_resolver.go:508 failed to push during intent resolution: failed to push id=4e28d923 key=/Local/Range/System/tsd/RangeDescriptor rw=true pri=0.00049351 stat=PENDING epo=0 ts=1567541572.400978423,0 orig=0.000000000,0 min=1567541572.400978423,0 max=0.000000000,0 wto=false seq=7 I190903 20:12:52.435491 4904 storage/store_snapshot.go:1038 [n2,replicate,s2,r4/2:/System{/tsd-tse}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 1953, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:52.440240 6240 storage/replica_raftstorage.go:793 [n3,s3,r4/3:{-}] applying LEARNER snapshot [id=40220d07 index=42] I190903 20:12:52.442540 114 gossip/gossip.go:1531 [n1] node has connected to cluster via gossip I190903 20:12:52.442677 114 storage/stores.go:259 [n1] wrote 2 node addresses to persistent storage I190903 20:12:52.443184 6240 storage/replica_raftstorage.go:814 [n3,s3,r4/3:/System{/tsd-tse}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=40220d07 index=42] I190903 20:12:52.444069 4904 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [(n3,s3):3] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=4] I190903 20:12:52.465002 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 53, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 20:12:52.467498 6229 storage/store.go:2593 [n1,replicaGC,s1,r4/1:/System{/tsd-tse}] removing replica r4/1 I190903 20:12:52.470096 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=54 (unpartitioned) version=3 mtime=2019-09-03 20:12:52.469949179 +0000 UTC I190903 20:12:52.470171 6229 storage/replica_destroy.go:152 [n1,replicaGC,s1,r4/1:/System{/tsd-tse}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:52.484447 4904 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 20:12:52.485060 4904 storage/replica_command.go:1430 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3, next=4, gen=5] I190903 20:12:52.514378 4904 storage/replica_raft.go:291 [n2,s2,r4/2:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:12:52.514429 4904 storage/replica_raft.go:304 [n2,s2,r4/2:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:12:52.520274 4904 storage/replica_command.go:979 [n2,replicate,s2,r4/2:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/2:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:12:52.520313 4904 storage/queue.go:1027 [n2,replicate,s2,r4/2:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/2:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:12:52.522471 6856 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2:0, 3*:48] I190903 20:12:52.522528 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 55, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:52.524023 6669 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/{Table/54-Max}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r21:/{Table/54-Max} [(n1,s1):1, (n2,s2):2, (n3,s3):3, next=4, gen=5] I190903 20:12:52.533547 6669 storage/replica_raft.go:291 [n1,s1,r21/1:/{Table/54-Max}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n2,s2):2] next=4 I190903 20:12:52.534805 6669 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/{Table/54-Max}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r21:/{Table/54-Max} [(n1,s1):1, (n2,s2):2, next=4, gen=6] I190903 20:12:52.541414 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 56, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 20:12:52.542640 7078 storage/replica_command.go:284 [n1,split,s1,r21/1:/{Table/54-Max}] initiating a split of this range at key /Table/56 [r22] (zone config) I190903 20:12:52.544519 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=56 (single col list partitioning) version=2 mtime=2019-09-03 20:12:52.543461733 +0000 UTC I190903 20:12:52.545534 6669 storage/replica_raft.go:291 [n1,s1,r21/1:/{Table/54-Max}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=4 I190903 20:12:52.557080 7078 storage/split_queue.go:149 [n1,split,s1,r21/1:/{Table/54-Max}] split saw concurrent descriptor modification; maybe retrying I190903 20:12:52.557191 7337 storage/replica_command.go:284 [n1,split,s1,r21/1:/{Table/54-Max}] initiating a split of this range at key /Table/56 [r23] (zone config) I190903 20:12:52.558325 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=56 (single col list partitioning) version=3 mtime=2019-09-03 20:12:52.558168791 +0000 UTC I190903 20:12:52.565169 6807 storage/store.go:2593 [n3,replicaGC,s3,r21/3:/{Table/54-Max}] removing replica r21/3 I190903 20:12:52.565312 6219 storage/store.go:2593 [n2,replicaGC,s2,r21/2:/{Table/54-Max}] removing replica r21/2 I190903 20:12:52.565420 6807 storage/replica_destroy.go:152 [n3,replicaGC,s3,r21/3:/{Table/54-Max}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:52.565592 6219 storage/replica_destroy.go:152 [n2,replicaGC,s2,r21/2:/{Table/54-Max}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:52.632534 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=56 (single col list partitioning) version=4 mtime=2019-09-03 20:12:52.629736066 +0000 UTC I190903 20:12:52.651930 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 56, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:52.678042 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 56, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:52.679621 7964 storage/replica_command.go:284 [n1,split,s1,r23/1:/{Table/56-Max}] initiating a split of this range at key /Table/56/1 [r24] (zone config) I190903 20:12:52.689010 8083 storage/replica_command.go:284 [n1,split,s1,r24/1:/{Table/56/1-Max}] initiating a split of this range at key /Table/56/2 [r25] (zone config) I190903 20:12:52.707210 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 56, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:52.708258 8173 storage/replica_command.go:284 [n1,split,s1,r24/1:/Table/56/{1-2}] initiating a split of this range at key /Table/56/1/3 [r26] (zone config) I190903 20:12:52.716655 8311 storage/replica_command.go:284 [n1,split,s1,r26/1:/Table/56/{1/3-2}] initiating a split of this range at key /Table/56/1/4 [r27] (zone config) I190903 20:12:52.720494 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 56, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:52.720681 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.731611 7407 storage/replica_command.go:284 [n1,split,s1,r27/1:/Table/56/{1/4-2}] initiating a split of this range at key /Table/56/1/5 [r28] (zone config) I190903 20:12:52.732132 8269 storage/replica_command.go:1430 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r26:/Table/56/1/{3-4} [(n1,s1):1, next=2, gen=12] I190903 20:12:52.732296 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.737693 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.740606 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.743197 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.746414 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.748663 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.755319 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.759149 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.759565 8269 storage/replica_raft.go:291 [n1,s1,r26/1:/Table/56/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:52.764581 8269 storage/store_snapshot.go:995 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] sending LEARNER snapshot d7f7db94 at applied index 17 I190903 20:12:52.764767 8269 storage/store_snapshot.go:1038 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.765626 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.765995 8574 storage/replica_raftstorage.go:793 [n2,s2,r26/2:{-}] applying LEARNER snapshot [id=d7f7db94 index=17] I190903 20:12:52.768217 8574 storage/replica_raftstorage.go:814 [n2,s2,r26/2:/Table/56/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d7f7db94 index=17] I190903 20:12:52.769174 8269 storage/replica_command.go:1430 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r26:/Table/56/1/{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=13] I190903 20:12:52.773380 8269 storage/replica_raft.go:291 [n1,s1,r26/1:/Table/56/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:52.774071 8269 storage/replica_command.go:1430 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r26:/Table/56/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=14] I190903 20:12:52.776075 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 20:12:52.777755 8395 storage/queue.go:1027 [n2,replicate,s2,r4/2:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2*:51, 3:0] I190903 20:12:52.783542 8269 storage/replica_raft.go:291 [n1,s1,r26/1:/Table/56/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:52.783593 8269 storage/replica_raft.go:304 [n1,s1,r26/1:/Table/56/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:52.784269 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.785802 8269 storage/replica_command.go:979 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r26 failed: storage/replica_raft.go:305: [n1,s1,r26/1:/Table/56/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:52.785875 8269 storage/queue.go:1027 [n1,replicate,s1,r26/1:/Table/56/1/{3-4}] change replicas of r26 failed: storage/replica_raft.go:305: [n1,s1,r26/1:/Table/56/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:52.786363 8577 storage/replica_command.go:1430 [n1,replicate,s1,r23/1:/Table/56{-/1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r23:/Table/56{-/1} [(n1,s1):1, next=2, gen=9] I190903 20:12:52.793496 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.795500 8577 storage/replica_raft.go:291 [n1,s1,r23/1:/Table/56{-/1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:52.796886 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 20:12:52.804393 8397 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2:0, 3*:53] I190903 20:12:52.806050 8577 storage/store_snapshot.go:995 [n1,replicate,s1,r23/1:/Table/56{-/1}] sending LEARNER snapshot 687dac61 at applied index 18 I190903 20:12:52.806176 8577 storage/store_snapshot.go:1038 [n1,replicate,s1,r23/1:/Table/56{-/1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.807333 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.807410 8747 storage/replica_raftstorage.go:793 [n3,s3,r23/2:{-}] applying LEARNER snapshot [id=687dac61 index=18] I190903 20:12:52.811236 8747 storage/replica_raftstorage.go:814 [n3,s3,r23/2:/Table/56{-/1}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=687dac61 index=18] I190903 20:12:52.815861 8577 storage/replica_command.go:1430 [n1,replicate,s1,r23/1:/Table/56{-/1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r23:/Table/56{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=10] I190903 20:12:52.816386 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.818637 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.820767 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.827108 8577 storage/replica_raft.go:291 [n1,s1,r23/1:/Table/56{-/1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:52.827623 8577 storage/replica_command.go:1430 [n1,replicate,s1,r23/1:/Table/56{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r23:/Table/56{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=11] I190903 20:12:52.830371 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.832951 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.839558 8577 storage/replica_raft.go:291 [n1,s1,r23/1:/Table/56{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:52.839605 8577 storage/replica_raft.go:304 [n1,s1,r23/1:/Table/56{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:52.840335 8577 storage/replica_command.go:979 [n1,replicate,s1,r23/1:/Table/56{-/1}] could not promote [] to voter, rolling back: change replicas of r23 failed: storage/replica_raft.go:305: [n1,s1,r23/1:/Table/56{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:52.840379 8577 storage/queue.go:1027 [n1,replicate,s1,r23/1:/Table/56{-/1}] change replicas of r23 failed: storage/replica_raft.go:305: [n1,s1,r23/1:/Table/56{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:52.844059 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.844202 8895 storage/replica_command.go:1430 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r27:/Table/56/1/{4-5} [(n1,s1):1, next=2, gen=13] I190903 20:12:52.845509 8865 storage/replica_command.go:1430 [n2,replicate,s2,r26/2:/Table/56/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r26:/Table/56/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=14] I190903 20:12:52.847154 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:52.848381 8895 storage/replica_raft.go:291 [n1,s1,r27/1:/Table/56/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:52.851859 8895 storage/store_snapshot.go:995 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] sending LEARNER snapshot 54d0330b at applied index 17 I190903 20:12:52.851983 8895 storage/store_snapshot.go:1038 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.852059 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.854548 8865 storage/replica_raft.go:291 [n2,s2,r26/2:/Table/56/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:52.856802 8931 storage/replica_raftstorage.go:793 [n3,s3,r27/2:{-}] applying LEARNER snapshot [id=54d0330b index=17] I190903 20:12:52.859035 8931 storage/replica_raftstorage.go:814 [n3,s3,r27/2:/Table/56/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=54d0330b index=17] I190903 20:12:52.860255 8895 storage/replica_command.go:1430 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r27:/Table/56/1/{4-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=14] I190903 20:12:52.900254 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.906478 8895 storage/replica_raft.go:291 [n1,s1,r27/1:/Table/56/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:52.911754 8895 storage/replica_command.go:1430 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r27:/Table/56/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=15] E190903 20:12:52.961734 9117 storage/queue.go:1027 [n2,replicate,s2,r4/2:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2*:56, 3:0] I190903 20:12:52.962025 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.964121 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.965458 8895 storage/replica_raft.go:291 [n1,s1,r27/1:/Table/56/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:52.965517 8895 storage/replica_raft.go:304 [n1,s1,r27/1:/Table/56/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:52.966472 8895 storage/replica_command.go:979 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r27 failed: storage/replica_raft.go:305: [n1,s1,r27/1:/Table/56/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:52.966520 8895 storage/queue.go:1027 [n1,replicate,s1,r27/1:/Table/56/1/{4-5}] change replicas of r27 failed: storage/replica_raft.go:305: [n1,s1,r27/1:/Table/56/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:52.966967 8983 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, next=4, gen=8] I190903 20:12:52.967082 8959 storage/store.go:2593 [n1,replicaGC,s1,r26/1:/Table/56/1/{3-4}] removing replica r26/1 I190903 20:12:52.967388 8959 storage/replica_destroy.go:152 [n1,replicaGC,s1,r26/1:/Table/56/1/{3-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:52.969516 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.971420 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.971611 8983 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n1,s1):1 (n2,s2):4LEARNER] next=5 I190903 20:12:52.974197 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.976976 8983 storage/store_snapshot.go:995 [n1,replicate,s1,r21/1:/Table/5{4-6}] sending LEARNER snapshot d30f92f9 at applied index 21 I190903 20:12:52.977117 8983 storage/store_snapshot.go:1038 [n1,replicate,s1,r21/1:/Table/5{4-6}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:52.978471 9145 storage/replica_raftstorage.go:793 [n2,s2,r21/4:{-}] applying LEARNER snapshot [id=d30f92f9 index=21] I190903 20:12:52.978635 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.985958 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:52.988846 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:52.993907 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:52.994072 9145 storage/replica_raftstorage.go:814 [n2,s2,r21/4:/Table/5{4-6}] applied LEARNER snapshot [total=16ms ingestion=4@15ms id=d30f92f9 index=21] I190903 20:12:52.995126 8983 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [(n2,s2):4] remove []): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n2,s2):4LEARNER, next=5, gen=9] I190903 20:12:53.003718 8983 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n1,s1):1 (n2,s2):4] next=5 I190903 20:12:53.004221 8983 storage/replica_command.go:1430 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n2,s2):4, next=5, gen=10] I190903 20:12:53.004537 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.006456 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.008936 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.012798 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.013272 8983 storage/replica_raft.go:291 [n1,s1,r21/1:/Table/5{4-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):4] next=5 E190903 20:12:53.013318 8983 storage/replica_raft.go:304 [n1,s1,r21/1:/Table/5{4-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:12:53.017010 8983 storage/replica_command.go:979 [n1,replicate,s1,r21/1:/Table/5{4-6}] could not promote [] to voter, rolling back: change replicas of r21 failed: storage/replica_raft.go:305: [n1,s1,r21/1:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 20:12:53.017050 8983 storage/queue.go:1027 [n1,replicate,s1,r21/1:/Table/5{4-6}] change replicas of r21 failed: storage/replica_raft.go:305: [n1,s1,r21/1:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:12:53.020799 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.023294 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.025306 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.032214 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.035163 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.037540 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.047692 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.052905 2055 gossip/gossip.go:1531 [n3] node has connected to cluster via gossip I190903 20:12:53.053048 2055 storage/stores.go:259 [n3] wrote 2 node addresses to persistent storage I190903 20:12:53.054165 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.058376 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.063568 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.084829 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.089029 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.095379 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.109749 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.112072 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.119983 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:12:53.134100 9713 storage/queue.go:1027 [n2,replicate,s2,r21/4:/Table/5{4-6}] no removable replicas from range that needs a removal: [1:0, 4*:28] I190903 20:12:53.139308 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.141253 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.147686 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:12:53.152829 9805 storage/queue.go:1027 [n3,replicate,s3,r27/2:/Table/56/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:12:53.185871 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:12:53.244474 10048 storage/queue.go:1027 [n3,replicate,s3,r23/2:/Table/56{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:12:53.255350 9992 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] no removable replicas from range that needs a removal: [2:0, 3*:59] I190903 20:12:53.301870 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.304692 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.308104 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.346537 10070 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):2, (n3,s3):3, next=4, gen=5] I190903 20:12:53.348279 10382 storage/replica_command.go:1430 [n2,replicate,s2,r21/4:/Table/5{4-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r21:/Table/5{4-6} [(n1,s1):1, (n2,s2):4, next=5, gen=10] I190903 20:12:53.356923 10070 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 20:12:53.357842 10382 storage/replica_raft.go:291 [n2,s2,r21/4:/Table/5{4-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):4] next=5 I190903 20:12:53.359478 10395 storage/replica_command.go:1430 [n3,replicate,s3,r23/2:/Table/56{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r23:/Table/56{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=11] I190903 20:12:53.367245 10395 storage/replica_raft.go:291 [n3,s3,r23/2:/Table/56{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:53.420588 10274 storage/store.go:2593 [n1,replicaGC,s1,r21/1:/Table/5{4-6}] removing replica r21/1 I190903 20:12:53.420948 10274 storage/replica_destroy.go:152 [n1,replicaGC,s1,r21/1:/Table/5{4-6}] removed 11 (0+11) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:53.421237 10659 storage/store.go:2593 [n1,replicaGC,s1,r23/1:/Table/56{-/1}] removing replica r23/1 I190903 20:12:53.421535 10659 storage/replica_destroy.go:152 [n1,replicaGC,s1,r23/1:/Table/56{-/1}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:53.422012 10289 storage/store.go:2593 [n2,replicaGC,s2,r4/2:/System{/tsd-tse}] removing replica r4/2 I190903 20:12:53.422517 10289 storage/replica_destroy.go:152 [n2,replicaGC,s2,r4/2:/System{/tsd-tse}] removed 1951 (1941+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:53.447764 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:53.449736 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:53.453018 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:53.453942 10783 storage/replica_command.go:1430 [n3,replicate,s3,r27/2:/Table/56/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r27:/Table/56/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=15] I190903 20:12:53.457386 6180 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 === RUN TestRepartitioning/unpartitioned/single_col_list_partitioning_-_DEFAULT I190903 20:12:53.461707 10783 storage/replica_raft.go:291 [n3,s3,r27/2:/Table/56/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:53.471401 10413 storage/store.go:2593 [n1,replicaGC,s1,r27/1:/Table/56/1/{4-5}] removing replica r27/1 I190903 20:12:53.471704 10413 storage/replica_destroy.go:152 [n1,replicaGC,s1,r27/1:/Table/56/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:53.476472 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=56 (single col list partitioning) version=5 mtime=2019-09-03 20:12:53.459549643 +0000 UTC I190903 20:12:53.490438 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 55, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 20:12:53.494832 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 56 4}] I190903 20:12:53.548347 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=56 (single col list partitioning) version=6 mtime=2019-09-03 20:12:53.548210204 +0000 UTC I190903 20:12:53.580943 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 57, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:53.589709 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 58, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 20:12:53.590905 11664 storage/replica_command.go:284 [n1,split,s1,r25/1:/{Table/56/2-Max}] initiating a split of this range at key /Table/58 [r29] (zone config) I190903 20:12:53.591093 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=2 mtime=2019-09-03 20:12:53.590058294 +0000 UTC I190903 20:12:53.614553 1374 gossip/gossip.go:1531 [n2] node has connected to cluster via gossip I190903 20:12:53.614700 1374 storage/stores.go:259 [n2] wrote 2 node addresses to persistent storage I190903 20:12:53.625867 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=58 (single col list partitioning - DEFAULT) version=3 mtime=2019-09-03 20:12:53.625700465 +0000 UTC I190903 20:12:53.631508 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=4 mtime=2019-09-03 20:12:53.630977635 +0000 UTC I190903 20:12:53.637599 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 58, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:ALTER TABLE "single col list partitioning - DEFAULT" PARTITION BY LIST (a) (PARTITION p4 VALUES IN ((4)), PARTITION p5 VALUES IN ((5)), PARTITION pd VALUES IN ((DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:53.655714 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:53.656868 12364 storage/replica_command.go:284 [n1,split,s1,r29/1:/{Table/58-Max}] initiating a split of this range at key /Table/58/1/4 [r30] (zone config) I190903 20:12:53.664642 12475 storage/replica_command.go:284 [n1,split,s1,r30/1:/{Table/58/1/4-Max}] initiating a split of this range at key /Table/58/1/5 [r31] (zone config) I190903 20:12:53.683806 12513 storage/replica_command.go:284 [n1,split,s1,r31/1:/{Table/58/1/5-Max}] initiating a split of this range at key /Table/58/1/6 [r32] (zone config) I190903 20:12:53.685795 12541 storage/replica_command.go:1430 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r30:/Table/58/1/{4-5} [(n1,s1):1, next=2, gen=13] I190903 20:12:53.688763 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:53.700672 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 58, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:53.701096 12145 storage/replica_command.go:284 [n1,split,s1,r29/1:/Table/58{-/1/4}] initiating a split of this range at key /Table/58/1 [r33] (zone config) I190903 20:12:53.701423 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.713652 12541 storage/replica_raft.go:291 [n1,s1,r30/1:/Table/58/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:53.714655 12541 storage/store_snapshot.go:995 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] sending LEARNER snapshot dfb2c451 at applied index 18 I190903 20:12:53.714775 12541 storage/store_snapshot.go:1038 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:53.715937 12932 storage/replica_raftstorage.go:793 [n2,s2,r30/2:{-}] applying LEARNER snapshot [id=dfb2c451 index=18] I190903 20:12:53.716120 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.719058 12622 storage/replica_command.go:284 [n1,split,s1,r32/1:/{Table/58/1/6-Max}] initiating a split of this range at key /Table/58/2 [r34] (zone config) I190903 20:12:53.723811 12932 storage/replica_raftstorage.go:814 [n2,s2,r30/2:/Table/58/1/{4-5}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=dfb2c451 index=18] I190903 20:12:53.724091 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.725363 12541 storage/replica_command.go:1430 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r30:/Table/58/1/{4-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=14] I190903 20:12:53.728404 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.728433 12541 storage/replica_raft.go:291 [n1,s1,r30/1:/Table/58/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:53.730878 12541 storage/replica_command.go:1430 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r30:/Table/58/1/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=15] I190903 20:12:53.736051 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.742072 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:53.744010 12541 storage/replica_raft.go:291 [n1,s1,r30/1:/Table/58/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:53.744053 12541 storage/replica_raft.go:304 [n1,s1,r30/1:/Table/58/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:53.744979 12541 storage/replica_command.go:979 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r30 failed: storage/replica_raft.go:305: [n1,s1,r30/1:/Table/58/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:53.745065 12541 storage/queue.go:1027 [n1,replicate,s1,r30/1:/Table/58/1/{4-5}] change replicas of r30 failed: storage/replica_raft.go:305: [n1,s1,r30/1:/Table/58/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:53.745645 13051 storage/replica_command.go:1430 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):1, next=2, gen=11] I190903 20:12:53.749400 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.752361 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.753955 13051 storage/replica_raft.go:291 [n1,s1,r25/1:/Table/5{6/2-8}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:53.762126 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:53.763200 13051 storage/store_snapshot.go:995 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] sending LEARNER snapshot fc8a02f9 at applied index 19 I190903 20:12:53.763349 13051 storage/store_snapshot.go:1038 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:53.764003 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.764575 12942 storage/replica_raftstorage.go:793 [n2,s2,r25/2:{-}] applying LEARNER snapshot [id=fc8a02f9 index=19] I190903 20:12:53.765852 12942 storage/replica_raftstorage.go:814 [n2,s2,r25/2:/Table/5{6/2-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=fc8a02f9 index=19] I190903 20:12:53.766432 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.766598 13051 storage/replica_command.go:1430 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] change replicas (add [(n2,s2):2] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=12] I190903 20:12:53.768643 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:53.773138 13051 storage/replica_raft.go:291 [n1,s1,r25/1:/Table/5{6/2-8}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:53.773663 13051 storage/replica_command.go:1430 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):1, (n2,s2):2, next=3, gen=13] I190903 20:12:53.774040 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:53.776667 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.778816 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.789444 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.796975 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.800005 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.801301 13051 storage/replica_raft.go:291 [n1,s1,r25/1:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:53.802189 13051 storage/replica_raft.go:304 [n1,s1,r25/1:/Table/5{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:53.805456 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.808858 13051 storage/replica_command.go:979 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] could not promote [] to voter, rolling back: change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/1:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:53.819075 13051 storage/queue.go:1027 [n1,replicate,s1,r25/1:/Table/5{6/2-8}] change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/1:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:53.819671 13252 storage/replica_command.go:1430 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r31:/Table/58/1/{5-6} [(n1,s1):1, next=2, gen=14] I190903 20:12:53.820416 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.831687 13252 storage/replica_raft.go:291 [n1,s1,r31/1:/Table/58/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:53.832256 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:53.837855 13252 storage/store_snapshot.go:995 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] sending LEARNER snapshot 6c293e2d at applied index 18 I190903 20:12:53.838064 13252 storage/store_snapshot.go:1038 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:53.839148 13222 storage/replica_raftstorage.go:793 [n3,s3,r31/2:{-}] applying LEARNER snapshot [id=6c293e2d index=18] I190903 20:12:53.840390 13222 storage/replica_raftstorage.go:814 [n3,s3,r31/2:/Table/58/1/{5-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=6c293e2d index=18] I190903 20:12:53.840767 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:53.841595 13252 storage/replica_command.go:1430 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] change replicas (add [(n3,s3):2] remove []): existing descriptor r31:/Table/58/1/{5-6} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=15] I190903 20:12:53.846240 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.847105 13252 storage/replica_raft.go:291 [n1,s1,r31/1:/Table/58/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:53.847597 13252 storage/replica_command.go:1430 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r31:/Table/58/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=16] I190903 20:12:53.849231 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.854128 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.858226 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.860280 13252 storage/replica_raft.go:291 [n1,s1,r31/1:/Table/58/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:53.860738 13252 storage/replica_raft.go:304 [n1,s1,r31/1:/Table/58/1/{5-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:53.861596 13252 storage/replica_command.go:979 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] could not promote [] to voter, rolling back: change replicas of r31 failed: storage/replica_raft.go:305: [n1,s1,r31/1:/Table/58/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:53.861641 13252 storage/queue.go:1027 [n1,replicate,s1,r31/1:/Table/58/1/{5-6}] change replicas of r31 failed: storage/replica_raft.go:305: [n1,s1,r31/1:/Table/58/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:53.862063 13414 storage/replica_command.go:1430 [n1,replicate,s1,r29/1:/Table/58{-/1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r29:/Table/58{-/1} [(n1,s1):1, next=2, gen=13] I190903 20:12:53.865113 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.865910 13414 storage/replica_raft.go:291 [n1,s1,r29/1:/Table/58{-/1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:53.867675 13414 storage/store_snapshot.go:995 [n1,replicate,s1,r29/1:/Table/58{-/1}] sending LEARNER snapshot 4bb9d460 at applied index 21 I190903 20:12:53.868452 13414 storage/store_snapshot.go:1038 [n1,replicate,s1,r29/1:/Table/58{-/1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:53.874011 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.875878 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.879682 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.882191 13089 storage/replica_raftstorage.go:793 [n3,s3,r29/2:{-}] applying LEARNER snapshot [id=4bb9d460 index=21] I190903 20:12:53.883823 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.888026 13089 storage/replica_raftstorage.go:814 [n3,s3,r29/2:/Table/58{-/1}] applied LEARNER snapshot [total=3ms ingestion=4@3ms id=4bb9d460 index=21] I190903 20:12:53.889420 13414 storage/replica_command.go:1430 [n1,replicate,s1,r29/1:/Table/58{-/1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r29:/Table/58{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=14] I190903 20:12:53.893389 13414 storage/replica_raft.go:291 [n1,s1,r29/1:/Table/58{-/1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:53.893904 13414 storage/replica_command.go:1430 [n1,replicate,s1,r29/1:/Table/58{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r29:/Table/58{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=15] I190903 20:12:53.894352 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.897132 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.908706 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.911413 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.914418 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.920107 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.927634 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.930523 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.937321 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.942297 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.945670 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.949154 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.953622 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.955777 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.963399 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:53.974618 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:53.986235 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.006462 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:54.012454 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.053022 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:54.055029 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.124120 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:54.126005 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.262142 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:54.264019 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.534462 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:54.536700 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:54.851910 13414 storage/replica_raft.go:291 [n1,s1,r29/1:/Table/58{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:54.851955 13414 storage/replica_raft.go:304 [n1,s1,r29/1:/Table/58{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:54.852753 13414 storage/replica_command.go:979 [n1,replicate,s1,r29/1:/Table/58{-/1}] could not promote [] to voter, rolling back: change replicas of r29 failed: storage/replica_raft.go:305: [n1,s1,r29/1:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:54.852794 13414 storage/queue.go:1027 [n1,replicate,s1,r29/1:/Table/58{-/1}] change replicas of r29 failed: storage/replica_raft.go:305: [n1,s1,r29/1:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:54.856620 16004 storage/queue.go:1027 [n2,replicate,s2,r30/2:/Table/58/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:26] E190903 20:12:54.857068 13573 storage/queue.go:1027 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] no removable replicas from range that needs a removal: [1:0, 2*:28] E190903 20:12:54.857857 16036 storage/queue.go:1027 [n3,replicate,s3,r31/2:/Table/58/1/{5-6}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:12:54.858480 16037 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r34:/{Table/58/2-Max} [(n1,s1):1, next=2, gen=15] E190903 20:12:54.858764 15922 storage/queue.go:1027 [n3,replicate,s3,r29/2:/Table/58{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:28] I190903 20:12:54.862165 16037 storage/replica_raft.go:291 [n1,s1,r34/1:/{Table/58/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:54.863603 16037 storage/store_snapshot.go:995 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] sending LEARNER snapshot 833e3016 at applied index 15 I190903 20:12:54.863748 16037 storage/store_snapshot.go:1038 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:54.864969 13104 storage/replica_raftstorage.go:793 [n2,s2,r34/2:{-}] applying LEARNER snapshot [id=833e3016 index=15] I190903 20:12:54.866292 13104 storage/replica_raftstorage.go:814 [n2,s2,r34/2:/{Table/58/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=833e3016 index=15] I190903 20:12:54.873087 16037 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r34:/{Table/58/2-Max} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=16] I190903 20:12:54.876439 16037 storage/replica_raft.go:291 [n1,s1,r34/1:/{Table/58/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:54.876909 16037 storage/replica_command.go:1430 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r34:/{Table/58/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=17] I190903 20:12:54.886117 16037 storage/replica_raft.go:291 [n1,s1,r34/1:/{Table/58/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:54.886164 16037 storage/replica_raft.go:304 [n1,s1,r34/1:/{Table/58/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:54.887465 16037 storage/replica_command.go:979 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] could not promote [] to voter, rolling back: change replicas of r34 failed: storage/replica_raft.go:305: [n1,s1,r34/1:/{Table/58/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:54.887505 16037 storage/queue.go:1027 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] change replicas of r34 failed: storage/replica_raft.go:305: [n1,s1,r34/1:/{Table/58/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:54.914750 13481 storage/replica_command.go:1430 [n2,replicate,s2,r30/2:/Table/58/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r30:/Table/58/1/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=15] I190903 20:12:54.927150 13481 storage/replica_raft.go:291 [n2,s2,r30/2:/Table/58/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:54.993224 16104 storage/store.go:2593 [n1,replicaGC,s1,r30/1:/Table/58/1/{4-5}] removing replica r30/1 I190903 20:12:54.993587 16104 storage/replica_destroy.go:152 [n1,replicaGC,s1,r30/1:/Table/58/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.006679 16476 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):1, (n2,s2):2, next=3, gen=13] I190903 20:12:55.014846 16476 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:55.075812 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:55.079826 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:55.079853 16546 storage/store.go:2593 [n1,replicaGC,s1,r25/1:/Table/5{6/2-8}] removing replica r25/1 I190903 20:12:55.080119 16546 storage/replica_destroy.go:152 [n1,replicaGC,s1,r25/1:/Table/5{6/2-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.084049 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:55.085973 10713 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 === RUN TestRepartitioning/unpartitioned/single_col_range_partitioning I190903 20:12:55.099092 16169 storage/replica_command.go:1430 [n3,replicate,s3,r31/2:/Table/58/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r31:/Table/58/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=16] I190903 20:12:55.111089 16169 storage/replica_raft.go:291 [n3,s3,r31/2:/Table/58/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:55.126976 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=58 (single col list partitioning - DEFAULT) version=5 mtime=2019-09-03 20:12:55.088563508 +0000 UTC I190903 20:12:55.127307 16845 storage/replica_command.go:1430 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r29:/Table/58{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=15] I190903 20:12:55.127445 16274 storage/store.go:2593 [n1,replicaGC,s1,r31/1:/Table/58/1/{5-6}] removing replica r31/1 I190903 20:12:55.127742 16274 storage/replica_destroy.go:152 [n1,replicaGC,s1,r31/1:/Table/58/1/{5-6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 20:12:55.130540 16875 storage/queue.go:1027 [n2,replicate,s2,r34/2:/{Table/58/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 20:12:55.145678 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 57, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning - DEFAULT"]} I190903 20:12:55.164437 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 58 4}] I190903 20:12:55.172899 16845 storage/replica_raft.go:291 [n3,s3,r29/2:/Table/58{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:55.184095 16922 storage/store.go:2593 [n1,replicaGC,s1,r29/1:/Table/58{-/1}] removing replica r29/1 I190903 20:12:55.184416 16922 storage/replica_destroy.go:152 [n1,replicaGC,s1,r29/1:/Table/58{-/1}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.223776 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=58 (single col list partitioning - DEFAULT) version=6 mtime=2019-09-03 20:12:55.22258455 +0000 UTC E190903 20:12:55.243679 17429 storage/queue.go:1027 [n1,replicate,s1,r34/1:/{Table/58/2-Max}] no removable replicas from range that needs a removal: [1*:26, 2:0] I190903 20:12:55.253091 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 59, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:55.262857 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 60, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 20:12:55.264108 17357 storage/replica_command.go:284 [n1,split,s1,r34/1:/{Table/58/2-Max}] initiating a split of this range at key /Table/60 [r35] (zone config) I190903 20:12:55.264203 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=60 (single col range partitioning) version=2 mtime=2019-09-03 20:12:55.263242865 +0000 UTC E190903 20:12:55.283251 17790 storage/queue.go:1027 [n2,replicate,s2,r34/2:/Table/{58/2-60}] no removable replicas from range that needs a removal: [1:0, 2*:33] I190903 20:12:55.284974 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=60 (single col range partitioning) version=3 mtime=2019-09-03 20:12:55.284822784 +0000 UTC I190903 20:12:55.291498 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=60 (single col range partitioning) version=4 mtime=2019-09-03 20:12:55.290861073 +0000 UTC I190903 20:12:55.297644 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 60, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:55.300434 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore E190903 20:12:55.316285 18106 storage/queue.go:1027 [n1,replicate,s1,r35/1:/{Table/60-Max}] no removable replicas from range that needs a removal: [1*:11, 2:0] I190903 20:12:55.322311 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 60, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:55.324203 18681 storage/replica_command.go:284 [n1,split,s1,r35/1:/{Table/60-Max}] initiating a split of this range at key /Table/60/1 [r36] (zone config) I190903 20:12:55.376223 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 60, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:55.385889 18561 storage/replica_command.go:284 [n1,split,s1,r36/1:/{Table/60/1-Max}] initiating a split of this range at key /Table/60/1/3 [r37] (zone config) I190903 20:12:55.386750 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 60, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:55.386861 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 E190903 20:12:55.398680 16802 storage/queue.go:1027 [n2,replicate,s2,r35/2:/Table/60{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:16] I190903 20:12:55.400299 19066 storage/replica_command.go:284 [n1,split,s1,r37/1:/{Table/60/1/3-Max}] initiating a split of this range at key /Table/60/1/4 [r38] (zone config) I190903 20:12:55.400461 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 20:12:55.406962 18765 storage/queue.go:1027 [n2,replicate,s2,r36/2:/Table/60/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:17] I190903 20:12:55.420085 19395 storage/replica_command.go:284 [n1,split,s1,r38/1:/{Table/60/1/4-Max}] initiating a split of this range at key /Table/60/2 [r39] (zone config) I190903 20:12:55.420660 19148 storage/replica_command.go:1430 [n2,replicate,s2,r21/4:/Table/5{4-6}] change replicas (add [(n3,s3):5LEARNER] remove []): existing descriptor r21:/Table/5{4-6} [(n2,s2):4, next=5, gen=11] I190903 20:12:55.421776 19383 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r37:/Table/60/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=21] I190903 20:12:55.427306 19148 storage/replica_raft.go:291 [n2,s2,r21/4:/Table/5{4-6}] proposing ADD_REPLICA[(n3,s3):5LEARNER]: after=[(n2,s2):4 (n3,s3):5LEARNER] next=6 I190903 20:12:55.430416 19148 storage/store_snapshot.go:995 [n2,replicate,s2,r21/4:/Table/5{4-6}] sending LEARNER snapshot 1e51ef4e at applied index 40 I190903 20:12:55.430578 19148 storage/store_snapshot.go:1038 [n2,replicate,s2,r21/4:/Table/5{4-6}] streamed snapshot to (n3,s3):5LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:55.431919 19309 storage/replica_raftstorage.go:793 [n3,s3,r21/5:{-}] applying LEARNER snapshot [id=1e51ef4e index=40] I190903 20:12:55.440658 19309 storage/replica_raftstorage.go:814 [n3,s3,r21/5:/Table/5{4-6}] applied LEARNER snapshot [total=9ms ingestion=4@8ms id=1e51ef4e index=40] I190903 20:12:55.443201 19148 storage/replica_command.go:1430 [n2,replicate,s2,r21/4:/Table/5{4-6}] change replicas (add [(n3,s3):5] remove []): existing descriptor r21:/Table/5{4-6} [(n2,s2):4, (n3,s3):5LEARNER, next=6, gen=12] I190903 20:12:55.444447 19383 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/60/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 I190903 20:12:55.447409 19383 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r37:/Table/60/1/{3-4} [(n1,s1):1, next=3, gen=22] I190903 20:12:55.453634 18976 storage/store.go:2593 [n2,replicaGC,s2,r37/2:/Table/60/1/{3-4}] removing replica r37/2 I190903 20:12:55.454136 18976 storage/replica_destroy.go:152 [n2,replicaGC,s2,r37/2:/Table/60/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.457966 19383 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/60/1/{3-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 20:12:55.459316 19148 storage/replica_raft.go:291 [n2,s2,r21/4:/Table/5{4-6}] proposing ADD_REPLICA[(n3,s3):5]: after=[(n2,s2):4 (n3,s3):5] next=6 I190903 20:12:55.459606 19383 storage/store_snapshot.go:995 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] sending LEARNER snapshot 3df7d16f at applied index 19 I190903 20:12:55.459796 19383 storage/store_snapshot.go:1038 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:55.459857 19148 storage/replica_command.go:1430 [n2,replicate,s2,r21/4:/Table/5{4-6}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r21:/Table/5{4-6} [(n2,s2):4, (n3,s3):5, next=6, gen=13] I190903 20:12:55.482990 19507 storage/replica_raftstorage.go:793 [n3,s3,r37/3:{-}] applying LEARNER snapshot [id=3df7d16f index=19] I190903 20:12:55.484516 19507 storage/replica_raftstorage.go:814 [n3,s3,r37/3:/Table/60/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=3df7d16f index=19] I190903 20:12:55.538964 19383 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r37:/Table/60/1/{3-4} [(n1,s1):1, (n3,s3):3LEARNER, next=4, gen=23] I190903 20:12:55.544409 19383 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/60/1/{3-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 20:12:55.544976 19383 storage/replica_command.go:1430 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r37:/Table/60/1/{3-4} [(n1,s1):1, (n3,s3):3, next=4, gen=24] I190903 20:12:55.545521 19148 storage/replica_raft.go:291 [n2,s2,r21/4:/Table/5{4-6}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 E190903 20:12:55.545555 19148 storage/replica_raft.go:304 [n2,s2,r21/4:/Table/5{4-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) I190903 20:12:55.547036 19148 storage/replica_command.go:979 [n2,replicate,s2,r21/4:/Table/5{4-6}] could not promote [] to voter, rolling back: change replicas of r21 failed: storage/replica_raft.go:305: [n2,s2,r21/4:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) E190903 20:12:55.547066 19148 storage/queue.go:1027 [n2,replicate,s2,r21/4:/Table/5{4-6}] change replicas of r21 failed: storage/replica_raft.go:305: [n2,s2,r21/4:/Table/5{4-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) E190903 20:12:55.549141 19520 storage/queue.go:1027 [n3,replicate,s3,r21/5:/Table/5{4-6}] no removable replicas from range that needs a removal: [4:0, 5*:47] I190903 20:12:55.566200 19397 storage/replica_command.go:1430 [n3,replicate,s3,r21/5:/Table/5{4-6}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r21:/Table/5{4-6} [(n2,s2):4, (n3,s3):5, next=6, gen=13] I190903 20:12:55.569629 19383 storage/replica_raft.go:291 [n1,s1,r37/1:/Table/60/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 E190903 20:12:55.572894 19383 storage/replica_raft.go:304 [n1,s1,r37/1:/Table/60/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:12:55.570271 19472 storage/replica_command.go:1430 [n2,replicate,s2,r35/2:/Table/60{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r35:/Table/60{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=19] I190903 20:12:55.577114 19383 storage/replica_command.go:979 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r37 failed: storage/replica_raft.go:305: [n1,s1,r37/1:/Table/60/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:12:55.577156 19383 storage/queue.go:1027 [n1,replicate,s1,r37/1:/Table/60/1/{3-4}] change replicas of r37 failed: storage/replica_raft.go:305: [n1,s1,r37/1:/Table/60/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:12:55.580200 19650 storage/replica_command.go:1430 [n1,replicate,s1,r38/1:/Table/60/{1/4-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r38:/Table/60/{1/4-2} [(n1,s1):1, (n2,s2):2, next=3, gen=22] I190903 20:12:55.583839 19397 storage/replica_raft.go:291 [n3,s3,r21/5:/Table/5{4-6}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 I190903 20:12:55.583934 19650 storage/replica_raft.go:291 [n1,s1,r38/1:/Table/60/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):1] next=3 E190903 20:12:55.588333 19846 storage/queue.go:1027 [n3,replicate,s3,r37/3:/Table/60/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 3*:25] I190903 20:12:55.589324 19472 storage/replica_raft.go:291 [n2,s2,r35/2:/Table/60{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:55.611168 19849 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/{Table/60/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r39:/{Table/60/2-Max} [(n1,s1):1, (n2,s2):2, next=3, gen=22] I190903 20:12:55.611293 19603 storage/store.go:2593 [n2,replicaGC,s2,r38/2:/Table/60/{1/4-2}] removing replica r38/2 I190903 20:12:55.611572 19603 storage/replica_destroy.go:152 [n2,replicaGC,s2,r38/2:/Table/60/{1/4-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.612364 20010 storage/store.go:2593 [n2,replicaGC,s2,r21/4:/Table/5{4-6}] removing replica r21/4 I190903 20:12:55.612739 20010 storage/replica_destroy.go:152 [n2,replicaGC,s2,r21/4:/Table/5{4-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.616428 19874 storage/store.go:2593 [n1,replicaGC,s1,r35/1:/Table/60{-/1}] removing replica r35/1 I190903 20:12:55.616688 19874 storage/replica_destroy.go:152 [n1,replicaGC,s1,r35/1:/Table/60{-/1}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.617704 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:12:55.630702 19849 storage/replica_raft.go:291 [n2,s2,r39/2:/{Table/60/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:55.631666 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:12:55.632468 20107 storage/replica_command.go:1430 [n2,replicate,s2,r36/2:/Table/60/1{-/3}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r36:/Table/60/1{-/3} [(n1,s1):1, (n2,s2):2, next=3, gen=20] I190903 20:12:55.643460 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:12:55.644396 19916 storage/store.go:2593 [n1,replicaGC,s1,r39/1:/{Table/60/2-Max}] removing replica r39/1 I190903 20:12:55.644611 19916 storage/replica_destroy.go:152 [n1,replicaGC,s1,r39/1:/{Table/60/2-Max}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.649142 16741 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:12:55.650046 20107 storage/replica_raft.go:291 [n2,s2,r36/2:/Table/60/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 === RUN TestRepartitioning/unpartitioned/single_col_range_partitioning_-_MAXVALUE I190903 20:12:55.658205 20056 storage/store.go:2593 [n1,replicaGC,s1,r36/1:/Table/60/1{-/3}] removing replica r36/1 I190903 20:12:55.658664 20056 storage/replica_destroy.go:152 [n1,replicaGC,s1,r36/1:/Table/60/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.671989 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=60 (single col range partitioning) version=5 mtime=2019-09-03 20:12:55.652311066 +0000 UTC I190903 20:12:55.686616 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 59, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 20:12:55.688550 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 60 4}] I190903 20:12:55.689750 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:12:55.721593 20509 storage/replica_command.go:1430 [n2,replicate,s2,r34/2:/Table/{58/2-60}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r34:/Table/{58/2-60} [(n1,s1):1, (n2,s2):2, next=3, gen=18] I190903 20:12:55.734314 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=60 (single col range partitioning) version=6 mtime=2019-09-03 20:12:55.734149437 +0000 UTC I190903 20:12:55.735848 20509 storage/replica_raft.go:291 [n2,s2,r34/2:/Table/{58/2-60}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:55.745370 20580 storage/store.go:2593 [n1,replicaGC,s1,r34/1:/Table/{58/2-60}] removing replica r34/1 I190903 20:12:55.745974 20580 storage/replica_destroy.go:152 [n1,replicaGC,s1,r34/1:/Table/{58/2-60}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:55.775125 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 61, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:55.776833 20659 storage/replica_command.go:1430 [n3,replicate,s3,r37/3:/Table/60/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r37:/Table/60/1/{3-4} [(n1,s1):1, (n3,s3):3, next=4, gen=24] I190903 20:12:55.783272 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 62, info: {TableName:data.public.unpartitioned Statement:CREATE TABLE unpartitioned (a INT8 PRIMARY KEY) User:root} I190903 20:12:55.784341 20659 storage/replica_raft.go:291 [n3,s3,r37/3:/Table/60/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 20:12:55.784944 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 20:12:55.783785916 +0000 UTC I190903 20:12:55.795065 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=62 (single col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 20:12:55.794920477 +0000 UTC I190903 20:12:55.795468 21220 storage/replica_command.go:284 [n2,split,s2,r39/2:/{Table/60/2-Max}] initiating a split of this range at key /Table/62 [r51] (zone config) I190903 20:12:55.809800 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 20:12:55.803118189 +0000 UTC I190903 20:12:55.818440 21449 storage/store.go:2593 [n1,replicaGC,s1,r37/1:/Table/60/1/{3-4}] removing replica r37/1 I190903 20:12:55.818949 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 62, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:ALTER TABLE "single col range partitioning - MAXVALUE" PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:55.822559 21449 storage/replica_destroy.go:152 [n1,replicaGC,s1,r37/1:/Table/60/1/{3-4}] removed 10 (0+10) keys in 4ms [clear=0ms commit=4ms] I190903 20:12:55.867893 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:55.878851 21966 storage/replica_command.go:284 [n2,split,s2,r51/1:/{Table/62-Max}] initiating a split of this range at key /Table/62/1 [r52] (zone config) I190903 20:12:55.900850 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:55.922696 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 62, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:55.925117 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:55.933765 22386 storage/replica_command.go:284 [n2,split,s2,r52/1:/{Table/62/1-Max}] initiating a split of this range at key /Table/62/1/4 [r53] (zone config) I190903 20:12:55.954290 22683 storage/replica_command.go:284 [n2,split,s2,r53/1:/{Table/62/1/4-Max}] initiating a split of this range at key /Table/62/1/5 [r54] (zone config) I190903 20:12:55.954916 22176 storage/replica_command.go:1430 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r52:/Table/62/1{-/4} [(n2,s2):1, next=2, gen=26] I190903 20:12:55.962674 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:55.969382 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:55.973558 22176 storage/replica_raft.go:291 [n2,s2,r52/1:/Table/62/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:12:55.974760 22821 storage/replica_command.go:284 [n2,split,s2,r54/1:/{Table/62/1/5-Max}] initiating a split of this range at key /Table/62/2 [r55] (zone config) I190903 20:12:55.977283 22176 storage/store_snapshot.go:995 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] sending LEARNER snapshot 186596b5 at applied index 17 I190903 20:12:55.977464 22176 storage/store_snapshot.go:1038 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:55.983099 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:55.987316 22593 storage/replica_raftstorage.go:793 [n1,s1,r52/2:{-}] applying LEARNER snapshot [id=186596b5 index=17] I190903 20:12:55.988517 22593 storage/replica_raftstorage.go:814 [n1,s1,r52/2:/Table/62/1{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=186596b5 index=17] I190903 20:12:55.988961 22176 storage/replica_command.go:1430 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r52:/Table/62/1{-/4} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=27] I190903 20:12:55.993401 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.001462 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.013568 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.020888 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.023242 22176 storage/replica_raft.go:291 [n2,s2,r52/1:/Table/62/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:12:56.023943 22176 storage/replica_command.go:1430 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r52:/Table/62/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=28] I190903 20:12:56.029096 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.041533 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.045291 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.045697 22176 storage/replica_raft.go:291 [n2,s2,r52/1:/Table/62/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:12:56.045782 22176 storage/replica_raft.go:304 [n2,s2,r52/1:/Table/62/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:56.046696 22176 storage/replica_command.go:979 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] could not promote [] to voter, rolling back: change replicas of r52 failed: storage/replica_raft.go:305: [n2,s2,r52/1:/Table/62/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:56.046728 22176 storage/queue.go:1027 [n2,replicate,s2,r52/1:/Table/62/1{-/4}] change replicas of r52 failed: storage/replica_raft.go:305: [n2,s2,r52/1:/Table/62/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:56.047249 23005 storage/replica_command.go:1430 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r55:/{Table/62/2-Max} [(n2,s2):1, next=2, gen=28] I190903 20:12:56.059949 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.063966 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.067384 23005 storage/replica_raft.go:291 [n2,s2,r55/1:/{Table/62/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:12:56.068783 23005 storage/store_snapshot.go:995 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] sending LEARNER snapshot 282fd44e at applied index 14 I190903 20:12:56.068969 23005 storage/store_snapshot.go:1038 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:56.080510 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.082550 23197 storage/replica_raftstorage.go:793 [n1,s1,r55/2:{-}] applying LEARNER snapshot [id=282fd44e index=14] I190903 20:12:56.089135 23197 storage/replica_raftstorage.go:814 [n1,s1,r55/2:/{Table/62/2-Max}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=282fd44e index=14] I190903 20:12:56.089903 23005 storage/replica_command.go:1430 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r55:/{Table/62/2-Max} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=29] I190903 20:12:56.096325 23005 storage/replica_raft.go:291 [n2,s2,r55/1:/{Table/62/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:12:56.096930 23005 storage/replica_command.go:1430 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r55:/{Table/62/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=30] I190903 20:12:56.102862 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.106828 23005 storage/replica_raft.go:291 [n2,s2,r55/1:/{Table/62/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:12:56.106878 23005 storage/replica_raft.go:304 [n2,s2,r55/1:/{Table/62/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:56.108435 23005 storage/replica_command.go:979 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] could not promote [] to voter, rolling back: change replicas of r55 failed: storage/replica_raft.go:305: [n2,s2,r55/1:/{Table/62/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:56.108732 23005 storage/queue.go:1027 [n2,replicate,s2,r55/1:/{Table/62/2-Max}] change replicas of r55 failed: storage/replica_raft.go:305: [n2,s2,r55/1:/{Table/62/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:56.109262 23208 storage/replica_command.go:1430 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r54:/Table/62/{1/5-2} [(n2,s2):1, next=2, gen=28] I190903 20:12:56.113129 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.122158 23208 storage/replica_raft.go:291 [n2,s2,r54/1:/Table/62/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:12:56.123195 23208 storage/store_snapshot.go:995 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] sending LEARNER snapshot 23f5ff9e at applied index 17 I190903 20:12:56.123378 23208 storage/store_snapshot.go:1038 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:56.123789 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.124812 23312 storage/replica_raftstorage.go:793 [n3,s3,r54/2:{-}] applying LEARNER snapshot [id=23f5ff9e index=17] I190903 20:12:56.132263 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.133801 23312 storage/replica_raftstorage.go:814 [n3,s3,r54/2:/Table/62/{1/5-2}] applied LEARNER snapshot [total=9ms ingestion=4@9ms id=23f5ff9e index=17] I190903 20:12:56.134394 23208 storage/replica_command.go:1430 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r54:/Table/62/{1/5-2} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=29] I190903 20:12:56.143149 23208 storage/replica_raft.go:291 [n2,s2,r54/1:/Table/62/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:12:56.143693 23208 storage/replica_command.go:1430 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r54:/Table/62/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=30] I190903 20:12:56.144302 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.147839 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.157169 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.160433 23208 storage/replica_raft.go:291 [n2,s2,r54/1:/Table/62/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:12:56.160500 23208 storage/replica_raft.go:304 [n2,s2,r54/1:/Table/62/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:56.161808 23208 storage/replica_command.go:979 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r54 failed: storage/replica_raft.go:305: [n2,s2,r54/1:/Table/62/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:56.161847 23208 storage/queue.go:1027 [n2,replicate,s2,r54/1:/Table/62/{1/5-2}] change replicas of r54 failed: storage/replica_raft.go:305: [n2,s2,r54/1:/Table/62/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:56.162279 23638 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r25:/Table/5{6/2-8} [(n2,s2):2, next=3, gen=14] I190903 20:12:56.162554 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.166110 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.172129 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.174609 23638 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/5{6/2-8}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:12:56.176131 23638 storage/store_snapshot.go:995 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] sending LEARNER snapshot 14ba238b at applied index 38 I190903 20:12:56.176401 23638 storage/store_snapshot.go:1038 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:56.176422 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.177258 23551 storage/replica_raftstorage.go:793 [n1,s1,r25/3:{-}] applying LEARNER snapshot [id=14ba238b index=38] I190903 20:12:56.180446 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.198900 23551 storage/replica_raftstorage.go:814 [n1,s1,r25/3:/Table/5{6/2-8}] applied LEARNER snapshot [total=22ms ingestion=4@21ms id=14ba238b index=38] I190903 20:12:56.201157 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.205060 23638 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] change replicas (add [(n1,s1):3] remove []): existing descriptor r25:/Table/5{6/2-8} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=15] I190903 20:12:56.241350 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.245205 23638 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/5{6/2-8}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:12:56.245796 23638 storage/replica_command.go:1430 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r25:/Table/5{6/2-8} [(n2,s2):2, (n1,s1):3, next=4, gen=16] I190903 20:12:56.256318 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.268429 23638 storage/replica_raft.go:291 [n2,s2,r25/2:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:12:56.268481 23638 storage/replica_raft.go:304 [n2,s2,r25/2:/Table/5{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:12:56.278429 23638 storage/replica_command.go:979 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] could not promote [] to voter, rolling back: change replicas of r25 failed: storage/replica_raft.go:305: [n2,s2,r25/2:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:12:56.278476 23638 storage/queue.go:1027 [n2,replicate,s2,r25/2:/Table/5{6/2-8}] change replicas of r25 failed: storage/replica_raft.go:305: [n2,s2,r25/2:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:12:56.283115 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.284115 24100 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, next=3, gen=24] E190903 20:12:56.284610 23827 storage/queue.go:1027 [n3,replicate,s3,r54/2:/Table/62/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:12:56.286387 23983 storage/queue.go:1027 [n1,replicate,s1,r55/2:/{Table/62/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 20:12:56.300372 24100 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:12:56.301424 24100 storage/store_snapshot.go:995 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] sending LEARNER snapshot 9043da7a at applied index 24 I190903 20:12:56.301620 24100 storage/store_snapshot.go:1038 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:56.302688 23871 storage/replica_raftstorage.go:793 [n1,s1,r39/3:{-}] applying LEARNER snapshot [id=9043da7a index=24] I190903 20:12:56.304048 23871 storage/replica_raftstorage.go:814 [n1,s1,r39/3:/Table/6{0/2-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9043da7a index=24] I190903 20:12:56.304727 24100 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=25] I190903 20:12:56.319412 24100 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:12:56.320001 24100 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n1,s1):3, next=4, gen=26] I190903 20:12:56.322221 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.333846 24100 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:12:56.333887 24100 storage/replica_raft.go:304 [n2,s2,r39/2:/Table/6{0/2-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:12:56.335045 24100 storage/replica_command.go:979 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] could not promote [] to voter, rolling back: change replicas of r39 failed: storage/replica_raft.go:305: [n2,s2,r39/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:12:56.335088 24100 storage/queue.go:1027 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas of r39 failed: storage/replica_raft.go:305: [n2,s2,r39/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:12:56.340175 24063 storage/queue.go:1027 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] no removable replicas from range that needs a removal: [2:0, 3*:45] E190903 20:12:56.343728 24336 storage/queue.go:1027 [n1,replicate,s1,r52/2:/Table/62/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:12:56.388431 24431 storage/replica_command.go:1430 [n1,replicate,s1,r55/2:/{Table/62/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r55:/{Table/62/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=30] I190903 20:12:56.392022 24431 storage/replica_raft.go:291 [n1,s1,r55/2:/{Table/62/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:12:56.395062 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.399436 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.404274 24295 storage/store.go:2593 [n2,replicaGC,s2,r55/1:/{Table/62/2-Max}] removing replica r55/1 I190903 20:12:56.404618 24295 storage/replica_destroy.go:152 [n2,replicaGC,s2,r55/1:/{Table/62/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:56.418397 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.492913 24777 storage/replica_command.go:1430 [n1,replicate,s1,r52/2:/Table/62/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r52:/Table/62/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=28] I190903 20:12:56.499087 24777 storage/replica_raft.go:291 [n1,s1,r52/2:/Table/62/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:12:56.508072 24781 storage/store.go:2593 [n2,replicaGC,s2,r52/1:/Table/62/1{-/4}] removing replica r52/1 I190903 20:12:56.508538 24781 storage/replica_destroy.go:152 [n2,replicaGC,s2,r52/1:/Table/62/1{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 20:12:56.556782 24954 storage/queue.go:1027 [n1,replicate,s1,r39/3:/Table/6{0/2-2}] no removable replicas from range that needs a removal: [2:0, 3*:32] I190903 20:12:56.604067 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.606167 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.609439 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:12:56.621888 25023 storage/replica_command.go:1430 [n3,replicate,s3,r54/2:/Table/62/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r54:/Table/62/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=30] I190903 20:12:56.629627 25023 storage/replica_raft.go:291 [n3,s3,r54/2:/Table/62/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 20:12:56.639607 25163 storage/store.go:2593 [n2,replicaGC,s2,r54/1:/Table/62/{1/5-2}] removing replica r54/1 I190903 20:12:56.640129 25163 storage/replica_destroy.go:152 [n2,replicaGC,s2,r54/1:/Table/62/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:56.703620 25268 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r25:/Table/5{6/2-8} [(n2,s2):2, (n1,s1):3, next=4, gen=16] I190903 20:12:56.706821 25268 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:12:56.709849 25441 storage/queue.go:1027 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] no removable replicas from range that needs a removal: [2*:35, 3:0] I190903 20:12:56.715442 25100 storage/store.go:2593 [n2,replicaGC,s2,r25/2:/Table/5{6/2-8}] removing replica r25/2 I190903 20:12:56.715853 25100 storage/replica_destroy.go:152 [n2,replicaGC,s2,r25/2:/Table/5{6/2-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:56.884736 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:12:56.886745 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:12:56.890069 20182 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 === RUN TestRepartitioning/single_col_list_partitioning/single_col_list_partitioning_-_DEFAULT I190903 20:12:56.903111 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=62 (single col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 20:12:56.893632579 +0000 UTC I190903 20:12:56.910580 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 61, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning - MAXVALUE"]} I190903 20:12:56.912550 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 62 4}] I190903 20:12:56.961976 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=62 (single col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 20:12:56.961815331 +0000 UTC I190903 20:12:56.982602 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 63, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:56.987602 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 64, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 20:12:56.989711 26850 storage/replica_command.go:284 [n1,split,s1,r55/2:/{Table/62/2-Max}] initiating a split of this range at key /Table/64 [r40] (zone config) I190903 20:12:56.998123 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:56.998755 26717 storage/replica_command.go:284 [n1,split,s1,r40/1:/{Table/64-Max}] initiating a split of this range at key /Table/64/1 [r41] (zone config) I190903 20:12:57.012940 26736 storage/replica_command.go:284 [n1,split,s1,r41/1:/{Table/64/1-Max}] initiating a split of this range at key /Table/64/2 [r42] (zone config) I190903 20:12:57.016467 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:57.025238 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:57.025387 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.029086 27472 storage/replica_command.go:284 [n1,split,s1,r41/1:/Table/64/{1-2}] initiating a split of this range at key /Table/64/1/3 [r43] (zone config) I190903 20:12:57.030419 27121 storage/replica_command.go:1430 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r42:/{Table/64/2-Max} [(n1,s1):1, next=2, gen=34] I190903 20:12:57.035269 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.038860 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.040595 27121 storage/replica_raft.go:291 [n1,s1,r42/1:/{Table/64/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:57.041275 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.041736 27121 storage/store_snapshot.go:995 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] sending LEARNER snapshot f0c6bc71 at applied index 14 I190903 20:12:57.041904 27121 storage/store_snapshot.go:1038 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.042483 27605 storage/replica_command.go:284 [n1,split,s1,r43/1:/Table/64/{1/3-2}] initiating a split of this range at key /Table/64/1/4 [r44] (zone config) I190903 20:12:57.043122 27599 storage/replica_raftstorage.go:793 [n3,s3,r42/2:{-}] applying LEARNER snapshot [id=f0c6bc71 index=14] I190903 20:12:57.043845 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.044595 27599 storage/replica_raftstorage.go:814 [n3,s3,r42/2:/{Table/64/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=f0c6bc71 index=14] I190903 20:12:57.046063 27121 storage/replica_command.go:1430 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r42:/{Table/64/2-Max} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=35] I190903 20:12:57.046459 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.048697 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.051146 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.052920 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.054158 27121 storage/replica_raft.go:291 [n1,s1,r42/1:/{Table/64/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:57.054584 27121 storage/replica_command.go:1430 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r42:/{Table/64/2-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=36] I190903 20:12:57.054881 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:57.055410 27702 storage/replica_command.go:284 [n1,split,s1,r44/1:/Table/64/{1/4-2}] initiating a split of this range at key /Table/64/1/5 [r45] (zone config) I190903 20:12:57.058159 27606 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n1,s1):3, next=4, gen=26] I190903 20:12:57.064757 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.066932 27121 storage/replica_raft.go:291 [n1,s1,r42/1:/{Table/64/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:57.066997 27121 storage/replica_raft.go:304 [n1,s1,r42/1:/{Table/64/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.070477 27121 storage/replica_command.go:979 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] could not promote [] to voter, rolling back: change replicas of r42 failed: storage/replica_raft.go:305: [n1,s1,r42/1:/{Table/64/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:57.070520 27121 storage/queue.go:1027 [n1,replicate,s1,r42/1:/{Table/64/2-Max}] change replicas of r42 failed: storage/replica_raft.go:305: [n1,s1,r42/1:/{Table/64/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.070612 27606 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):2] next=4 I190903 20:12:57.071825 27783 storage/replica_command.go:1430 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r43:/Table/64/1/{3-4} [(n1,s1):1, next=2, gen=36] I190903 20:12:57.071868 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.075183 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.079193 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.081309 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.081610 27783 storage/replica_raft.go:291 [n1,s1,r43/1:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:57.081830 27767 storage/store.go:2593 [n1,replicaGC,s1,r39/3:/Table/6{0/2-2}] removing replica r39/3 I190903 20:12:57.083245 27767 storage/replica_destroy.go:152 [n1,replicaGC,s1,r39/3:/Table/6{0/2-2}] removed 5 (0+5) keys in 1ms [clear=0ms commit=1ms] I190903 20:12:57.083249 27783 storage/store_snapshot.go:995 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] sending LEARNER snapshot 00c9ba2f at applied index 17 I190903 20:12:57.083453 27783 storage/store_snapshot.go:1038 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.084689 27846 storage/replica_raftstorage.go:793 [n2,s2,r43/2:{-}] applying LEARNER snapshot [id=00c9ba2f index=17] I190903 20:12:57.085885 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.094133 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.109445 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.112346 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.112721 27846 storage/replica_raftstorage.go:814 [n2,s2,r43/2:/Table/64/1/{3-4}] applied LEARNER snapshot [total=27ms ingestion=4@27ms id=00c9ba2f index=17] I190903 20:12:57.115587 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.117423 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.120509 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.122811 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.126579 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.128452 27783 storage/replica_command.go:1430 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r43:/Table/64/1/{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=37] I190903 20:12:57.128737 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:57.135365 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.138408 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:57.140726 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:57.144598 27783 storage/replica_raft.go:291 [n1,s1,r43/1:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:57.145637 27783 storage/replica_command.go:1430 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r43:/Table/64/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=38] I190903 20:12:57.147194 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.155437 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.162955 27783 storage/replica_raft.go:291 [n1,s1,r43/1:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:57.163010 27783 storage/replica_raft.go:304 [n1,s1,r43/1:/Table/64/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:57.163744 27783 storage/replica_command.go:979 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r43 failed: storage/replica_raft.go:305: [n1,s1,r43/1:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:57.163801 27783 storage/queue.go:1027 [n1,replicate,s1,r43/1:/Table/64/1/{3-4}] change replicas of r43 failed: storage/replica_raft.go:305: [n1,s1,r43/1:/Table/64/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:57.164293 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:57.165070 28074 storage/replica_command.go:1430 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r44:/Table/64/1/{4-5} [(n1,s1):1, next=2, gen=37] I190903 20:12:57.169054 28074 storage/replica_raft.go:291 [n1,s1,r44/1:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:57.170662 28074 storage/store_snapshot.go:995 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] sending LEARNER snapshot 9a68f83e at applied index 18 I190903 20:12:57.170841 28074 storage/store_snapshot.go:1038 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.171461 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:57.172008 27536 storage/replica_raftstorage.go:793 [n3,s3,r44/2:{-}] applying LEARNER snapshot [id=9a68f83e index=18] I190903 20:12:57.173516 27536 storage/replica_raftstorage.go:814 [n3,s3,r44/2:/Table/64/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9a68f83e index=18] I190903 20:12:57.174021 28074 storage/replica_command.go:1430 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r44:/Table/64/1/{4-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=38] I190903 20:12:57.174399 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.177536 28074 storage/replica_raft.go:291 [n1,s1,r44/1:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:57.178117 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.178602 28074 storage/replica_command.go:1430 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r44:/Table/64/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=39] I190903 20:12:57.182321 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.185588 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.190473 28074 storage/replica_raft.go:291 [n1,s1,r44/1:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:57.190518 28074 storage/replica_raft.go:304 [n1,s1,r44/1:/Table/64/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.191195 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.191317 28074 storage/replica_command.go:979 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r44 failed: storage/replica_raft.go:305: [n1,s1,r44/1:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:57.191360 28074 storage/queue.go:1027 [n1,replicate,s1,r44/1:/Table/64/1/{4-5}] change replicas of r44 failed: storage/replica_raft.go:305: [n1,s1,r44/1:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.197238 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.203645 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.206323 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.217024 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.235831 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.241410 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.253613 27861 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/64{-/1}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r40:/Table/64{-/1} [(n1,s1):1, next=2, gen=33] I190903 20:12:57.262654 27861 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/64{-/1}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:57.263830 27861 storage/store_snapshot.go:995 [n1,replicate,s1,r40/1:/Table/64{-/1}] sending LEARNER snapshot c6f6857e at applied index 18 I190903 20:12:57.264242 27861 storage/store_snapshot.go:1038 [n1,replicate,s1,r40/1:/Table/64{-/1}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.265421 27865 storage/replica_raftstorage.go:793 [n2,s2,r40/2:{-}] applying LEARNER snapshot [id=c6f6857e index=18] I190903 20:12:57.272402 27865 storage/replica_raftstorage.go:814 [n2,s2,r40/2:/Table/64{-/1}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=c6f6857e index=18] I190903 20:12:57.273493 27861 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/64{-/1}] change replicas (add [(n2,s2):2] remove []): existing descriptor r40:/Table/64{-/1} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=34] I190903 20:12:57.277941 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.291123 27861 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/64{-/1}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:57.292609 27861 storage/replica_command.go:1430 [n1,replicate,s1,r40/1:/Table/64{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r40:/Table/64{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=35] I190903 20:12:57.295973 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.312656 27861 storage/replica_raft.go:291 [n1,s1,r40/1:/Table/64{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:57.312694 27861 storage/replica_raft.go:304 [n1,s1,r40/1:/Table/64{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:57.313642 27861 storage/replica_command.go:979 [n1,replicate,s1,r40/1:/Table/64{-/1}] could not promote [] to voter, rolling back: change replicas of r40 failed: storage/replica_raft.go:305: [n1,s1,r40/1:/Table/64{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:57.313674 27861 storage/queue.go:1027 [n1,replicate,s1,r40/1:/Table/64{-/1}] change replicas of r40 failed: storage/replica_raft.go:305: [n1,s1,r40/1:/Table/64{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:57.321720 28500 storage/queue.go:1027 [n2,replicate,s2,r43/2:/Table/64/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:24] E190903 20:12:57.373738 28572 storage/queue.go:1027 [n3,replicate,s3,r42/2:/{Table/64/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 20:12:57.377229 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.382328 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 20:12:57.382747 28643 storage/queue.go:1027 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:12:57.399278 28720 storage/replica_command.go:1430 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r44:/Table/64/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=39] I190903 20:12:57.399848 28727 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, next=4, gen=17] I190903 20:12:57.403535 28727 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n1,s1):3 (n3,s3):4LEARNER] next=5 I190903 20:12:57.404693 28727 storage/store_snapshot.go:995 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] sending LEARNER snapshot 3ede5462 at applied index 53 I190903 20:12:57.404855 28727 storage/store_snapshot.go:1038 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.406027 28696 storage/replica_raftstorage.go:793 [n3,s3,r25/4:{-}] applying LEARNER snapshot [id=3ede5462 index=53] I190903 20:12:57.407331 28696 storage/replica_raftstorage.go:814 [n3,s3,r25/4:/Table/5{6/2-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=3ede5462 index=53] I190903 20:12:57.410298 28720 storage/replica_raft.go:291 [n3,s3,r44/2:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:57.412693 28727 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [(n3,s3):4] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n3,s3):4LEARNER, next=5, gen=18] I190903 20:12:57.417696 28727 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n1,s1):3 (n3,s3):4] next=5 I190903 20:12:57.418609 28727 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n3,s3):4, next=5, gen=19] I190903 20:12:57.432015 28846 storage/replica_command.go:1430 [n2,replicate,s2,r43/2:/Table/64/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r43:/Table/64/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=38] I190903 20:12:57.432847 28727 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 E190903 20:12:57.432896 28727 storage/replica_raft.go:304 [n1,s1,r25/3:/Table/5{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:12:57.434174 28727 storage/replica_command.go:979 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] could not promote [] to voter, rolling back: change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/3:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 20:12:57.434217 28727 storage/queue.go:1027 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/3:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:12:57.443039 28846 storage/replica_raft.go:291 [n2,s2,r43/2:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:57.465765 28671 storage/store.go:2593 [n1,replicaGC,s1,r44/1:/Table/64/1/{4-5}] removing replica r44/1 I190903 20:12:57.466112 28671 storage/replica_destroy.go:152 [n1,replicaGC,s1,r44/1:/Table/64/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:57.466434 28935 storage/store.go:2593 [n1,replicaGC,s1,r43/1:/Table/64/1/{3-4}] removing replica r43/1 I190903 20:12:57.466731 28935 storage/replica_destroy.go:152 [n1,replicaGC,s1,r43/1:/Table/64/1/{3-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:57.530570 28832 storage/replica_command.go:1430 [n3,replicate,s3,r42/2:/{Table/64/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r42:/{Table/64/2-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=36] I190903 20:12:57.537432 28832 storage/replica_raft.go:291 [n3,s3,r42/2:/{Table/64/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:57.581101 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:57.583172 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:57.591054 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:57.594263 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:57.597474 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=2 mtime=2019-09-03 20:12:57.596373521 +0000 UTC I190903 20:12:57.604369 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 1}] I190903 20:12:57.609980 29418 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=2:1567541888.623802879,0, refcount=0 I190903 20:12:57.615235 29165 storage/store.go:2593 [n1,replicaGC,s1,r42/1:/{Table/64/2-Max}] removing replica r42/1 I190903 20:12:57.615602 29165 storage/replica_destroy.go:152 [n1,replicaGC,s1,r42/1:/{Table/64/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:12:57.653469 29367 storage/queue.go:1027 [n2,replicate,s2,r40/2:/Table/64{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:12:57.658631 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=64 (single col list partitioning - DEFAULT) version=3 mtime=2019-09-03 20:12:57.658468029 +0000 UTC I190903 20:12:57.662905 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 2}] I190903 20:12:57.663539 29602 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=3:1567541891.097464891,0, refcount=0 I190903 20:12:57.690283 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=4 mtime=2019-09-03 20:12:57.688101924 +0000 UTC I190903 20:12:57.695816 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 64, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:ALTER TABLE "single col list partitioning - DEFAULT" PARTITION BY LIST (a) (PARTITION p4 VALUES IN ((4)), PARTITION p5 VALUES IN ((5)), PARTITION pd VALUES IN ((DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:57.707704 29932 sql/lease.go:923 new lease: 64("single col list partitioning - DEFAULT") ver=4:1567541891.098464891,0, refcount=0 I190903 20:12:57.709246 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 3}] I190903 20:12:57.710027 30245 storage/replica_command.go:1430 [n2,s2,r43/2:/Table/64/1/{3-4}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r43:/Table/64/1/{3-4} [(n2,s2):2, next=3, gen=39] I190903 20:12:57.720650 30245 storage/replica_raft.go:291 [n2,s2,r43/2:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:12:57.721770 30245 storage/store_snapshot.go:995 [n2,s2,r43/2:/Table/64/1/{3-4}] sending LEARNER snapshot 6f674b35 at applied index 31 I190903 20:12:57.721958 30245 storage/store_snapshot.go:1038 [n2,s2,r43/2:/Table/64/1/{3-4}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.723071 30253 storage/replica_raftstorage.go:793 [n1,s1,r43/3:{-}] applying LEARNER snapshot [id=6f674b35 index=31] I190903 20:12:57.727833 30253 storage/replica_raftstorage.go:814 [n1,s1,r43/3:/Table/64/1/{3-4}] applied LEARNER snapshot [total=5ms ingestion=4@4ms id=6f674b35 index=31] I190903 20:12:57.730578 30245 storage/replica_command.go:1430 [n2,s2,r43/2:/Table/64/1/{3-4}] change replicas (add [(n1,s1):3] remove []): existing descriptor r43:/Table/64/1/{3-4} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=40] I190903 20:12:57.740188 30245 storage/replica_raft.go:291 [n2,s2,r43/2:/Table/64/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:12:57.741508 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} E190903 20:12:57.743168 29966 storage/queue.go:1027 [n1,replicate,s1,r43/3:/Table/64/1/{3-4}] no removable replicas from range that needs a removal: [2:0, 3*:35] I190903 20:12:57.750318 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:57.751176 30620 storage/replica_command.go:284 [n1,split,s1,r45/1:/Table/64/{1/5-2}] initiating a split of this range at key /Table/64/1/6 [r46] (zone config) I190903 20:12:57.759384 30860 storage/replica_command.go:1430 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r45:/Table/64/1/{5-6} [(n1,s1):1, next=2, gen=38] I190903 20:12:57.760942 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 64, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:57.761359 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:57.763783 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.767270 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.770758 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.772758 30860 storage/replica_raft.go:291 [n1,s1,r45/1:/Table/64/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:12:57.773891 30860 storage/store_snapshot.go:995 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] sending LEARNER snapshot c8713bef at applied index 18 I190903 20:12:57.774075 30860 storage/store_snapshot.go:1038 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.775130 31030 storage/replica_raftstorage.go:793 [n3,s3,r45/2:{-}] applying LEARNER snapshot [id=c8713bef index=18] I190903 20:12:57.775159 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.785909 31030 storage/replica_raftstorage.go:814 [n3,s3,r45/2:/Table/64/1/{5-6}] applied LEARNER snapshot [total=11ms ingestion=4@2ms id=c8713bef index=18] I190903 20:12:57.795321 30860 storage/replica_command.go:1430 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] change replicas (add [(n3,s3):2] remove []): existing descriptor r45:/Table/64/1/{5-6} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=39] I190903 20:12:57.799260 217 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:12:57.799261 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.800367 30860 storage/replica_raft.go:291 [n1,s1,r45/1:/Table/64/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:12:57.802077 29939 storage/replica_command.go:1430 [n1,s1,r43/3:/Table/64/1/{3-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r43:/Table/64/1/{3-4} [(n2,s2):2, (n1,s1):3, next=4, gen=41] I190903 20:12:57.802823 30860 storage/replica_command.go:1430 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r45:/Table/64/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=40] I190903 20:12:57.815651 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.819878 29939 storage/replica_raft.go:291 [n1,s1,r43/3:/Table/64/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:12:57.823466 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.823521 29368 storage/replica_command.go:609 [n1,merge,s1,r41/1:/Table/64/1{-/3}] initiating a merge of r43:/Table/64/1/{3-4} [(n1,s1):3, next=4, gen=42] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:12:57.825588 30860 storage/replica_raft.go:291 [n1,s1,r45/1:/Table/64/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:12:57.825634 30860 storage/replica_raft.go:304 [n1,s1,r45/1:/Table/64/1/{5-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.827675 30860 storage/replica_command.go:979 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] could not promote [] to voter, rolling back: change replicas of r45 failed: storage/replica_raft.go:305: [n1,s1,r45/1:/Table/64/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:12:57.827718 30860 storage/queue.go:1027 [n1,replicate,s1,r45/1:/Table/64/1/{5-6}] change replicas of r45 failed: storage/replica_raft.go:305: [n1,s1,r45/1:/Table/64/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:12:57.831209 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.836169 31253 storage/store.go:2593 [n2,replicaGC,s2,r43/2:/Table/64/1/{3-4}] removing replica r43/2 I190903 20:12:57.837954 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:57.838196 31253 storage/replica_destroy.go:152 [n2,replicaGC,s2,r43/2:/Table/64/1/{3-4}] removed 12 (0+12) keys in 2ms [clear=1ms commit=1ms] I190903 20:12:57.840520 165 storage/store.go:2593 [n1,s1,r41/1:/Table/64/1{-/3}] removing replica r43/3 I190903 20:12:57.841782 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.879896 31450 storage/replica_command.go:1430 [n2,replicate,s2,r40/2:/Table/64{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r40:/Table/64{-/1} [(n1,s1):1, (n2,s2):2, next=3, gen=35] I190903 20:12:57.885030 30802 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [] remove [(n3,s3):4]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n3,s3):4, next=5, gen=19] I190903 20:12:57.887075 31480 storage/replica_command.go:1430 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r44:/Table/64/1/{4-5} [(n3,s3):2, next=3, gen=40] I190903 20:12:57.888797 30802 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n3,s3):4]: after=[(n1,s1):3] next=5 I190903 20:12:57.893063 31450 storage/replica_raft.go:291 [n2,s2,r40/2:/Table/64{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:12:57.897321 31480 storage/replica_raft.go:291 [n3,s3,r44/2:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:12:57.897579 31469 storage/store.go:2593 [n3,replicaGC,s3,r25/4:/Table/5{6/2-8}] removing replica r25/4 I190903 20:12:57.897926 31469 storage/replica_destroy.go:152 [n3,replicaGC,s3,r25/4:/Table/5{6/2-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:57.898687 31480 storage/store_snapshot.go:995 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] sending LEARNER snapshot 66add8cb at applied index 32 I190903 20:12:57.898874 31480 storage/store_snapshot.go:1038 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:57.900106 31496 storage/replica_raftstorage.go:793 [n2,s2,r44/3:{-}] applying LEARNER snapshot [id=66add8cb index=32] I190903 20:12:57.908850 31496 storage/replica_raftstorage.go:814 [n2,s2,r44/3:/Table/64/1/{4-5}] applied LEARNER snapshot [total=9ms ingestion=4@8ms id=66add8cb index=32] I190903 20:12:57.911274 31480 storage/replica_command.go:1430 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r44:/Table/64/1/{4-5} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=41] I190903 20:12:57.914250 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.922920 31480 storage/replica_raft.go:291 [n3,s3,r44/2:/Table/64/1/{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 20:12:57.923457 31480 storage/replica_command.go:1430 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r44:/Table/64/1/{4-5} [(n3,s3):2, (n2,s2):3, next=4, gen=42] I190903 20:12:57.933702 31480 storage/replica_raft.go:291 [n3,s3,r44/2:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 E190903 20:12:57.933748 31480 storage/replica_raft.go:304 [n3,s3,r44/2:/Table/64/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:12:57.934635 31480 storage/replica_command.go:979 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r44 failed: storage/replica_raft.go:305: [n3,s3,r44/2:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:12:57.934676 31480 storage/queue.go:1027 [n3,replicate,s3,r44/2:/Table/64/1/{4-5}] change replicas of r44 failed: storage/replica_raft.go:305: [n3,s3,r44/2:/Table/64/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:12:57.935340 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.942489 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.954851 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:57.958154 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.960541 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.964083 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.964520 31498 storage/store.go:2593 [n1,replicaGC,s1,r40/1:/Table/64{-/1}] removing replica r40/1 I190903 20:12:57.964861 31498 storage/replica_destroy.go:152 [n1,replicaGC,s1,r40/1:/Table/64{-/1}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:57.966840 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.972809 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:57.975010 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 E190903 20:12:57.982618 31800 storage/queue.go:1027 [n3,replicate,s3,r45/2:/Table/64/1/{5-6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:12:57.983047 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:57.993349 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.052855 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.057208 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.059143 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.062372 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.064519 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.068107 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.069819 32137 storage/replica_command.go:1430 [n3,replicate,s3,r45/2:/Table/64/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r45:/Table/64/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=40] I190903 20:12:58.070361 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.074535 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.076766 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.076784 32137 storage/replica_raft.go:291 [n3,s3,r45/2:/Table/64/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:12:58.080878 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.083617 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.085060 32079 storage/store.go:2593 [n1,replicaGC,s1,r45/1:/Table/64/1/{5-6}] removing replica r45/1 I190903 20:12:58.085286 32079 storage/replica_destroy.go:152 [n1,replicaGC,s1,r45/1:/Table/64/1/{5-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:58.085316 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.090611 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.092491 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.099604 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.101810 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.114051 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.133746 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.135788 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.173077 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.176577 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.178363 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.180210 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 E190903 20:12:58.224387 32625 storage/queue.go:1027 [n2,replicate,s2,r44/3:/Table/64/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 20:12:58.254637 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.256653 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.258520 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.419617 33133 storage/replica_command.go:1430 [n2,replicate,s2,r44/3:/Table/64/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r44:/Table/64/1/{4-5} [(n3,s3):2, (n2,s2):3, next=4, gen=42] I190903 20:12:58.431040 33133 storage/replica_raft.go:291 [n2,s2,r44/3:/Table/64/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 20:12:58.439735 33274 storage/store.go:2593 [n3,replicaGC,s3,r44/2:/Table/64/1/{4-5}] removing replica r44/2 I190903 20:12:58.440123 33274 storage/replica_destroy.go:152 [n3,replicaGC,s3,r44/2:/Table/64/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:58.453796 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.455793 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.459737 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.467622 25550 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 === RUN TestRepartitioning/single_col_list_partitioning_-_DEFAULT/single_col_list_partitioning I190903 20:12:58.489119 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=64 (single col list partitioning - DEFAULT) version=5 mtime=2019-09-03 20:12:58.469524964 +0000 UTC I190903 20:12:58.505623 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 63, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning - DEFAULT"]} I190903 20:12:58.509100 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning - DEFAULT 64 4}] I190903 20:12:58.568066 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=64 (single col list partitioning - DEFAULT) version=6 mtime=2019-09-03 20:12:58.567901135 +0000 UTC I190903 20:12:58.593435 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 65, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:12:58.599857 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 66, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:CREATE TABLE "single col list partitioning - DEFAULT" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p4 VALUES IN (4), PARTITION p5 VALUES IN (5), PARTITION pd VALUES IN (DEFAULT)) User:root} I190903 20:12:58.602818 31667 storage/replica_command.go:284 [n3,split,s3,r42/2:/{Table/64/2-Max}] initiating a split of this range at key /Table/66 [r81] (zone config) I190903 20:12:58.609742 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:58.617619 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:58.625976 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:58.626225 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.630604 35089 storage/replica_command.go:284 [n3,split,s3,r81/1:/{Table/66-Max}] initiating a split of this range at key /Table/66/1 [r82] (zone config) I190903 20:12:58.640686 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.647345 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.648075 35175 storage/replica_command.go:284 [n3,split,s3,r82/1:/{Table/66/1-Max}] initiating a split of this range at key /Table/66/1/4 [r83] (zone config) I190903 20:12:58.655733 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.659346 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.664567 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.665538 35179 storage/replica_command.go:284 [n3,split,s3,r83/1:/{Table/66/1/4-Max}] initiating a split of this range at key /Table/66/1/5 [r84] (zone config) I190903 20:12:58.667585 35213 storage/replica_command.go:1430 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r82:/Table/66/1{-/4} [(n3,s3):1, next=2, gen=40] I190903 20:12:58.670558 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.678607 35113 storage/replica_command.go:284 [n3,split,s3,r84/1:/{Table/66/1/5-Max}] initiating a split of this range at key /Table/66/1/6 [r85] (zone config) I190903 20:12:58.681769 35213 storage/replica_raft.go:291 [n3,s3,r82/1:/Table/66/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:12:58.683155 35213 storage/store_snapshot.go:995 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] sending LEARNER snapshot 82635c8d at applied index 18 I190903 20:12:58.683342 35213 storage/store_snapshot.go:1038 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:58.684442 35400 storage/replica_raftstorage.go:793 [n1,s1,r82/2:{-}] applying LEARNER snapshot [id=82635c8d index=18] I190903 20:12:58.687087 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.689573 35400 storage/replica_raftstorage.go:814 [n1,s1,r82/2:/Table/66/1{-/4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=82635c8d index=18] I190903 20:12:58.692331 35213 storage/replica_command.go:1430 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r82:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=41] I190903 20:12:58.697167 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.702281 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.705612 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.707015 35545 storage/replica_command.go:284 [n3,split,s3,r85/1:/{Table/66/1/6-Max}] initiating a split of this range at key /Table/66/2 [r86] (zone config) I190903 20:12:58.711446 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.725629 35213 storage/replica_raft.go:291 [n3,s3,r82/1:/Table/66/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:12:58.728984 35213 storage/replica_command.go:1430 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r82:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=42] I190903 20:12:58.733541 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.738154 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.744862 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.747773 35213 storage/replica_raft.go:291 [n3,s3,r82/1:/Table/66/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:12:58.747820 35213 storage/replica_raft.go:304 [n3,s3,r82/1:/Table/66/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:58.748769 35213 storage/replica_command.go:979 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] could not promote [] to voter, rolling back: change replicas of r82 failed: storage/replica_raft.go:305: [n3,s3,r82/1:/Table/66/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:58.748809 35213 storage/queue.go:1027 [n3,replicate,s3,r82/1:/Table/66/1{-/4}] change replicas of r82 failed: storage/replica_raft.go:305: [n3,s3,r82/1:/Table/66/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:58.749184 35440 storage/replica_command.go:1430 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r83:/Table/66/1/{4-5} [(n3,s3):1, next=2, gen=41] I190903 20:12:58.750606 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.756070 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.759144 35440 storage/replica_raft.go:291 [n3,s3,r83/1:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:12:58.767717 35440 storage/store_snapshot.go:995 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] sending LEARNER snapshot c44bca66 at applied index 17 I190903 20:12:58.767909 35440 storage/store_snapshot.go:1038 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:58.769441 35635 storage/replica_raftstorage.go:793 [n2,s2,r83/2:{-}] applying LEARNER snapshot [id=c44bca66 index=17] I190903 20:12:58.771158 35635 storage/replica_raftstorage.go:814 [n2,s2,r83/2:/Table/66/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=c44bca66 index=17] I190903 20:12:58.771767 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.777000 35440 storage/replica_command.go:1430 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r83:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=42] I190903 20:12:58.785824 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.791135 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.792459 35440 storage/replica_raft.go:291 [n3,s3,r83/1:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:12:58.793518 35440 storage/replica_command.go:1430 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r83:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=43] I190903 20:12:58.795793 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.803867 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.807780 35440 storage/replica_raft.go:291 [n3,s3,r83/1:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:12:58.807874 35440 storage/replica_raft.go:304 [n3,s3,r83/1:/Table/66/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:58.808971 35440 storage/replica_command.go:979 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r83 failed: storage/replica_raft.go:305: [n3,s3,r83/1:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:58.809043 35440 storage/queue.go:1027 [n3,replicate,s3,r83/1:/Table/66/1/{4-5}] change replicas of r83 failed: storage/replica_raft.go:305: [n3,s3,r83/1:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:58.810172 35978 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, next=4, gen=6] I190903 20:12:58.811880 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:58.816816 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.822925 35978 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n3,s3):3 (n2,s2):4LEARNER] next=5 I190903 20:12:58.823539 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:58.824249 35978 storage/store_snapshot.go:995 [n3,replicate,s3,r4/3:/System{/tsd-tse}] sending LEARNER snapshot 952a1b10 at applied index 68 I190903 20:12:58.831464 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.832737 35978 storage/store_snapshot.go:1038 [n3,replicate,s3,r4/3:/System{/tsd-tse}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 1955, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:12:58.840279 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.844391 36071 storage/replica_raftstorage.go:793 [n2,s2,r4/4:{-}] applying LEARNER snapshot [id=952a1b10 index=68] I190903 20:12:58.854467 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.858079 36071 storage/replica_raftstorage.go:814 [n2,s2,r4/4:/System{/tsd-tse}] applied LEARNER snapshot [total=13ms ingestion=4@12ms id=952a1b10 index=68] I190903 20:12:58.858623 35978 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [(n2,s2):4] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4LEARNER, next=5, gen=7] I190903 20:12:58.867619 35978 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n3,s3):3 (n2,s2):4] next=5 I190903 20:12:58.868363 35978 storage/replica_command.go:1430 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4, next=5, gen=8] I190903 20:12:58.874798 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.878263 35978 storage/replica_raft.go:291 [n3,s3,r4/3:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 E190903 20:12:58.878297 35978 storage/replica_raft.go:304 [n3,s3,r4/3:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:12:58.879730 35978 storage/replica_command.go:979 [n3,replicate,s3,r4/3:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/3:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 20:12:58.879766 35978 storage/queue.go:1027 [n3,replicate,s3,r4/3:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/3:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:12:58.885065 36341 storage/replica_command.go:1430 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r86:/{Table/66/2-Max} [(n3,s3):1, next=2, gen=43] E190903 20:12:58.886828 36272 storage/queue.go:1027 [n1,replicate,s1,r82/2:/Table/66/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 20:12:58.893616 36341 storage/replica_raft.go:291 [n3,s3,r86/1:/{Table/66/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:12:58.894710 36341 storage/store_snapshot.go:995 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] sending LEARNER snapshot 67352df0 at applied index 15 I190903 20:12:58.894866 36341 storage/store_snapshot.go:1038 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:58.895999 36274 storage/replica_raftstorage.go:793 [n1,s1,r86/2:{-}] applying LEARNER snapshot [id=67352df0 index=15] I190903 20:12:58.897686 36274 storage/replica_raftstorage.go:814 [n1,s1,r86/2:/{Table/66/2-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=67352df0 index=15] I190903 20:12:58.898169 36341 storage/replica_command.go:1430 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r86:/{Table/66/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=44] I190903 20:12:58.906288 36341 storage/replica_raft.go:291 [n3,s3,r86/1:/{Table/66/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:12:58.907054 36341 storage/replica_command.go:1430 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r86:/{Table/66/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 20:12:58.910917 36452 storage/replica_command.go:1430 [n1,replicate,s1,r82/2:/Table/66/1{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r82:/Table/66/1{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=42] I190903 20:12:58.911850 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:58.914169 36452 storage/replica_raft.go:291 [n1,s1,r82/2:/Table/66/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:12:58.924486 36500 storage/store.go:2593 [n3,replicaGC,s3,r82/1:/Table/66/1{-/4}] removing replica r82/1 I190903 20:12:58.924799 36500 storage/replica_destroy.go:152 [n3,replicaGC,s3,r82/1:/Table/66/1{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:58.925535 36341 storage/replica_raft.go:291 [n3,s3,r86/1:/{Table/66/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:12:58.925581 36341 storage/replica_raft.go:304 [n3,s3,r86/1:/{Table/66/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:58.926403 36341 storage/replica_command.go:979 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] could not promote [] to voter, rolling back: change replicas of r86 failed: storage/replica_raft.go:305: [n3,s3,r86/1:/{Table/66/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:58.926602 36341 storage/queue.go:1027 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] change replicas of r86 failed: storage/replica_raft.go:305: [n3,s3,r86/1:/{Table/66/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:58.929532 36554 storage/queue.go:1027 [n1,replicate,s1,r86/2:/{Table/66/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 20:12:58.930037 36540 storage/replica_command.go:1430 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r85:/Table/66/{1/6-2} [(n3,s3):1, next=2, gen=43] I190903 20:12:58.940332 36540 storage/replica_raft.go:291 [n3,s3,r85/1:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:12:58.942831 36540 storage/store_snapshot.go:995 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] sending LEARNER snapshot 5c18c937 at applied index 20 I190903 20:12:58.942998 36540 storage/store_snapshot.go:1038 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:58.944131 36339 storage/replica_raftstorage.go:793 [n1,s1,r85/2:{-}] applying LEARNER snapshot [id=5c18c937 index=20] I190903 20:12:58.947781 36339 storage/replica_raftstorage.go:814 [n1,s1,r85/2:/Table/66/{1/6-2}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=5c18c937 index=20] I190903 20:12:58.956377 36540 storage/replica_command.go:1430 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r85:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=44] I190903 20:12:58.965347 36540 storage/replica_raft.go:291 [n3,s3,r85/1:/Table/66/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:12:58.965880 36540 storage/replica_command.go:1430 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r85:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 20:12:58.986091 36540 storage/replica_raft.go:291 [n3,s3,r85/1:/Table/66/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:12:58.986140 36540 storage/replica_raft.go:304 [n3,s3,r85/1:/Table/66/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:58.986892 36540 storage/replica_command.go:979 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r85 failed: storage/replica_raft.go:305: [n3,s3,r85/1:/Table/66/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:58.986934 36540 storage/queue.go:1027 [n3,replicate,s3,r85/1:/Table/66/{1/6-2}] change replicas of r85 failed: storage/replica_raft.go:305: [n3,s3,r85/1:/Table/66/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:58.987022 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:58.989412 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 E190903 20:12:58.994639 36626 storage/queue.go:1027 [n2,replicate,s2,r4/4:/System{/tsd-tse}] no removable replicas from range that needs a removal: [3:0, 4*:74] I190903 20:12:59.116937 37003 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [] remove [(n3,s3):3]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):3, (n2,s2):4, next=5, gen=8] I190903 20:12:59.127452 37003 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n3,s3):3]: after=[(n2,s2):4] next=5 I190903 20:12:59.129098 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:59.129287 37143 storage/replica_command.go:1430 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r83:/Table/66/1/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=43] I190903 20:12:59.131274 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:59.138316 37143 storage/replica_raft.go:291 [n2,s2,r83/2:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:12:59.167659 35616 storage/consistency_queue.go:133 [n3,consistencyChecker,s3,r83/1:/Table/66/1/{4-5}] could not get replica descriptor: r83 was not found on s3 E190903 20:12:59.167716 35616 storage/queue.go:1027 [n3,consistencyChecker,s3,r83/1:/Table/66/1/{4-5}] could not get replica descriptor: r83 was not found on s3 I190903 20:12:59.180737 36689 storage/store.go:2593 [n3,replicaGC,s3,r4/3:/System{/tsd-tse}] removing replica r4/3 I190903 20:12:59.181335 36689 storage/replica_destroy.go:152 [n3,replicaGC,s3,r4/3:/System{/tsd-tse}] removed 1950 (1941+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:59.182634 37365 storage/store.go:2593 [n3,replicaGC,s3,r83/1:/Table/66/1/{4-5}] removing replica r83/1 I190903 20:12:59.183032 37365 storage/replica_destroy.go:152 [n3,replicaGC,s3,r83/1:/Table/66/1/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 20:12:59.296001 37644 storage/queue.go:1027 [n1,replicate,s1,r85/2:/Table/66/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 20:12:59.354459 37456 storage/replica_command.go:1430 [n1,replicate,s1,r85/2:/Table/66/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r85:/Table/66/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 20:12:59.357848 37456 storage/replica_raft.go:291 [n1,s1,r85/2:/Table/66/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:12:59.360902 37865 storage/queue.go:1027 [n3,replicate,s3,r86/1:/{Table/66/2-Max}] no removable replicas from range that needs a removal: [1*:26, 2:0] I190903 20:12:59.366668 37198 storage/store.go:2593 [n3,replicaGC,s3,r85/1:/Table/66/{1/6-2}] removing replica r85/1 I190903 20:12:59.367035 37198 storage/replica_destroy.go:152 [n3,replicaGC,s3,r85/1:/Table/66/{1/6-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:12:59.405577 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:12:59.407585 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:12:59.411433 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:12:59.416950 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:12:59.421767 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=66 (single col list partitioning) version=2 mtime=2019-09-03 20:12:59.420459052 +0000 UTC I190903 20:12:59.430254 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 1}] I190903 20:12:59.433099 38183 sql/lease.go:923 new lease: 66("single col list partitioning") ver=2:1567541870.465697067,0, refcount=0 I190903 20:12:59.487985 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=66 (single col list partitioning) version=3 mtime=2019-09-03 20:12:59.487817544 +0000 UTC I190903 20:12:59.492052 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 2}] I190903 20:12:59.498140 38198 sql/lease.go:923 new lease: 66("single col list partitioning") ver=3:1567541884.262557872,0, refcount=0 I190903 20:12:59.518747 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=66 (single col list partitioning) version=4 mtime=2019-09-03 20:12:59.516580799 +0000 UTC I190903 20:12:59.523840 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 66, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:12:59.545348 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 3}] I190903 20:12:59.546133 38360 sql/lease.go:923 new lease: 66("single col list partitioning") ver=4:1567541884.263557872,0, refcount=0 E190903 20:12:59.554045 39000 storage/queue.go:1027 [n1,replicate,s1,r86/2:/{Table/66/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 20:12:59.578091 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:12:59.591336 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:12:59.591521 39355 storage/replica_command.go:284 [n1,split,s1,r82/2:/Table/66/1{-/4}] initiating a split of this range at key /Table/66/1/3 [r47] (zone config) E190903 20:12:59.595433 37696 storage/queue.go:1027 [n3,merge,s3,r81/1:/Table/66{-/1}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:12:59.596340 38641 storage/replica_command.go:1430 [n1,s1,r86/2:/{Table/66/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r86:/{Table/66/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=45] I190903 20:12:59.600571 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 66, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:12:59.600673 38641 storage/replica_raft.go:291 [n1,s1,r86/2:/{Table/66/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:12:59.600804 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:59.602637 38853 storage/replica_command.go:609 [n1,merge,s1,r85/2:/Table/66/{1/6-2}] initiating a merge of r86:/{Table/66/2-Max} [(n1,s1):2, next=3, gen=46] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:12:59.604016 39657 storage/replica_command.go:1430 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r47:/Table/66/1/{3-4} [(n1,s1):1, next=2, gen=44] I190903 20:12:59.607479 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.613231 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.617726 39225 storage/store.go:2593 [n3,replicaGC,s3,r86/1:/{Table/66/2-Max}] removing replica r86/1 I190903 20:12:59.618905 39225 storage/replica_destroy.go:152 [n3,replicaGC,s3,r86/1:/{Table/66/2-Max}] removed 8 (0+8) keys in 1ms [clear=1ms commit=0ms] I190903 20:12:59.620313 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.624032 39657 storage/replica_raft.go:291 [n1,s1,r47/1:/Table/66/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:12:59.626998 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.627688 38853 storage/replica_command.go:609 [n1,merge,s1,r85/2:/Table/66/{1/6-2}] initiating a merge of r86:/{Table/66/2-Max} [(n1,s1):2, next=3, gen=46] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:12:59.628519 39657 storage/store_snapshot.go:995 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] sending LEARNER snapshot 5036e7f3 at applied index 14 I190903 20:12:59.628820 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.629093 39657 storage/store_snapshot.go:1038 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:59.630232 39860 storage/replica_raftstorage.go:793 [n2,s2,r47/2:{-}] applying LEARNER snapshot [id=5036e7f3 index=14] I190903 20:12:59.632955 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.640420 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.645809 186 storage/store.go:2593 [n1,s1,r85/2:/Table/66/{1/6-2}] removing replica r86/2 I190903 20:12:59.651411 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.652208 39860 storage/replica_raftstorage.go:814 [n2,s2,r47/2:/Table/66/1/{3-4}] applied LEARNER snapshot [total=22ms ingestion=4@21ms id=5036e7f3 index=14] I190903 20:12:59.653015 39657 storage/replica_command.go:1430 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r47:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=45] I190903 20:12:59.654392 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.663567 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.664397 39657 storage/replica_raft.go:291 [n1,s1,r47/1:/Table/66/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:12:59.664917 39657 storage/replica_command.go:1430 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r47:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=46] I190903 20:12:59.674696 39657 storage/replica_raft.go:291 [n1,s1,r47/1:/Table/66/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:12:59.674755 39657 storage/replica_raft.go:304 [n1,s1,r47/1:/Table/66/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:12:59.677413 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.679753 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.679868 39657 storage/replica_command.go:979 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r47 failed: storage/replica_raft.go:305: [n1,s1,r47/1:/Table/66/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:59.679926 39657 storage/queue.go:1027 [n1,replicate,s1,r47/1:/Table/66/1/{3-4}] change replicas of r47 failed: storage/replica_raft.go:305: [n1,s1,r47/1:/Table/66/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:12:59.685842 40009 storage/queue.go:1027 [n2,replicate,s2,r47/2:/Table/66/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 20:12:59.747183 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:59.750977 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.756957 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.763344 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.766975 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.771528 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.773372 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.778161 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.782506 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.786946 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.793726 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.801475 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:59.807009 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.810110 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.815656 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.822542 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.825431 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.835103 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.840878 40586 storage/replica_command.go:1430 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r84:/Table/66/1/{5-6} [(n3,s3):1, next=2, gen=42] I190903 20:12:59.846793 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:59.851538 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:12:59.855883 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.856254 40586 storage/replica_raft.go:291 [n3,s3,r84/1:/Table/66/1/{5-6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:12:59.859660 40586 storage/store_snapshot.go:995 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] sending LEARNER snapshot c6d9104f at applied index 23 I190903 20:12:59.859772 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.859923 40586 storage/store_snapshot.go:1038 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:12:59.860948 40653 storage/replica_raftstorage.go:793 [n1,s1,r84/2:{-}] applying LEARNER snapshot [id=c6d9104f index=23] I190903 20:12:59.865789 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.867766 40653 storage/replica_raftstorage.go:814 [n1,s1,r84/2:/Table/66/1/{5-6}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=c6d9104f index=23] I190903 20:12:59.868321 40586 storage/replica_command.go:1430 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r84:/Table/66/1/{5-6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=43] I190903 20:12:59.872703 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.879196 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.882048 40586 storage/replica_raft.go:291 [n3,s3,r84/1:/Table/66/1/{5-6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:12:59.883040 40586 storage/replica_command.go:1430 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r84:/Table/66/1/{5-6} [(n3,s3):1, (n1,s1):2, next=3, gen=44] I190903 20:12:59.887995 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.889921 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.894809 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.902423 40586 storage/replica_raft.go:291 [n3,s3,r84/1:/Table/66/1/{5-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:12:59.902463 40586 storage/replica_raft.go:304 [n3,s3,r84/1:/Table/66/1/{5-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:59.903059 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.909310 40586 storage/replica_command.go:979 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] could not promote [] to voter, rolling back: change replicas of r84 failed: storage/replica_raft.go:305: [n3,s3,r84/1:/Table/66/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:12:59.909367 40586 storage/queue.go:1027 [n3,replicate,s3,r84/1:/Table/66/1/{5-6}] change replicas of r84 failed: storage/replica_raft.go:305: [n3,s3,r84/1:/Table/66/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:12:59.911955 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.928201 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.958778 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.961221 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.964318 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:12:59.976688 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:12:59.978688 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:12:59.982109 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:00.002743 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:00.009176 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:00.058388 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:00.131900 41542 storage/replica_command.go:1430 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r83:/Table/66/1/{4-5} [(n2,s2):2, next=3, gen=44] I190903 20:13:00.133500 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:00.139004 41542 storage/replica_raft.go:291 [n2,s2,r83/2:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:13:00.139999 41542 storage/store_snapshot.go:995 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] sending LEARNER snapshot 912f13f7 at applied index 34 I190903 20:13:00.140152 41542 storage/store_snapshot.go:1038 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:00.141576 40958 storage/replica_raftstorage.go:793 [n3,s3,r83/3:{-}] applying LEARNER snapshot [id=912f13f7 index=34] I190903 20:13:00.145742 40978 storage/replica_command.go:284 [n1,split,s1,r85/2:/{Table/66/1/6-Max}] initiating a split of this range at key /Table/66/2 [r48] (zone config) I190903 20:13:00.154450 40958 storage/replica_raftstorage.go:814 [n3,s3,r83/3:/Table/66/1/{4-5}] applied LEARNER snapshot [total=13ms ingestion=4@12ms id=912f13f7 index=34] I190903 20:13:00.155376 41542 storage/replica_command.go:1430 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r83:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=45] I190903 20:13:00.163267 41542 storage/replica_raft.go:291 [n2,s2,r83/2:/Table/66/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 20:13:00.163797 41542 storage/replica_command.go:1430 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r83:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3, next=4, gen=46] I190903 20:13:00.195187 41542 storage/replica_raft.go:291 [n2,s2,r83/2:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:13:00.195241 41542 storage/replica_raft.go:304 [n2,s2,r83/2:/Table/66/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:00.196157 41542 storage/replica_command.go:979 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r83 failed: storage/replica_raft.go:305: [n2,s2,r83/2:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:00.196205 41542 storage/queue.go:1027 [n2,replicate,s2,r83/2:/Table/66/1/{4-5}] change replicas of r83 failed: storage/replica_raft.go:305: [n2,s2,r83/2:/Table/66/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:00.196619 41813 storage/replica_command.go:1430 [n2,replicate,s2,r47/2:/Table/66/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r47:/Table/66/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=46] I190903 20:13:00.212683 41813 storage/replica_raft.go:291 [n2,s2,r47/2:/Table/66/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:00.216282 41583 storage/queue.go:1027 [n3,replicate,s3,r83/3:/Table/66/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 20:13:00.224202 41693 storage/store.go:2593 [n1,replicaGC,s1,r47/1:/Table/66/1/{3-4}] removing replica r47/1 I190903 20:13:00.224631 41693 storage/replica_destroy.go:152 [n1,replicaGC,s1,r47/1:/Table/66/1/{3-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:00.271978 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 20:13:00.273445 41603 storage/queue.go:1027 [n1,replicate,s1,r84/2:/Table/66/1/{5-6}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 20:13:00.275617 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:13:00.385233 42378 storage/queue.go:1027 [n1,merge,s1,r84/2:/Table/66/1/{5-6}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:00.444104 42379 storage/replica_command.go:1430 [n1,replicate,s1,r84/2:/Table/66/1/{5-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r84:/Table/66/1/{5-6} [(n3,s3):1, (n1,s1):2, next=3, gen=44] I190903 20:13:00.447665 42379 storage/replica_raft.go:291 [n1,s1,r84/2:/Table/66/1/{5-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:00.455702 42256 storage/store.go:2593 [n3,replicaGC,s3,r84/1:/Table/66/1/{5-6}] removing replica r84/1 I190903 20:13:00.456011 42256 storage/replica_destroy.go:152 [n3,replicaGC,s3,r84/1:/Table/66/1/{5-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:00.517387 42583 storage/replica_command.go:1430 [n3,replicate,s3,r83/3:/Table/66/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r83:/Table/66/1/{4-5} [(n2,s2):2, (n3,s3):3, next=4, gen=46] I190903 20:13:00.526208 42583 storage/replica_raft.go:291 [n3,s3,r83/3:/Table/66/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 20:13:00.534934 42737 storage/store.go:2593 [n2,replicaGC,s2,r83/2:/Table/66/1/{4-5}] removing replica r83/2 I190903 20:13:00.535297 42737 storage/replica_destroy.go:152 [n2,replicaGC,s2,r83/2:/Table/66/1/{4-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:00.599864 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:00.605545 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:00.607714 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:00.609950 33388 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 === RUN TestRepartitioning/multi_col_list_partitioning/multi_col_list_partitioning_-_DEFAULT I190903 20:13:00.622352 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=66 (single col list partitioning) version=5 mtime=2019-09-03 20:13:00.614043906 +0000 UTC I190903 20:13:00.630176 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 65, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 20:13:00.632265 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 66 4}] I190903 20:13:00.634001 43169 storage/replica_command.go:609 [n1,merge,s1,r84/2:/Table/66/1/{5-6}] initiating a merge of r85:/Table/66/{1/6-2} [(n1,s1):2, next=3, gen=48] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:00.642409 207 storage/store.go:2593 [n1,s1,r84/2:/Table/66/1/{5-6}] removing replica r85/2 I190903 20:13:00.686817 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=66 (single col list partitioning) version=6 mtime=2019-09-03 20:13:00.686607199 +0000 UTC I190903 20:13:00.705759 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 67, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:00.713204 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 68, info: {TableName:data.public."multi col list partitioning" Statement:CREATE TABLE "multi col list partitioning" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p56 VALUES IN ((5, 6)), PARTITION p57 VALUES IN ((5, 7))) User:root} I190903 20:13:00.713789 43585 storage/replica_command.go:284 [n1,split,s1,r48/1:/{Table/66/2-Max}] initiating a split of this range at key /Table/68 [r49] (zone config) I190903 20:13:00.719554 298 server/status/runtime.go:498 [n1] runtime stats: 218 MiB RSS, 961 goroutines, 63 MiB/41 MiB/138 MiB GO alloc/idle/total, 53 MiB/97 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (29x), 104 MiB/104 MiB (r/w)net I190903 20:13:00.723359 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:00.725582 44314 storage/replica_command.go:284 [n1,split,s1,r49/1:/{Table/68-Max}] initiating a split of this range at key /Table/68/1 [r50] (zone config) I190903 20:13:00.731784 44094 storage/replica_command.go:284 [n1,split,s1,r50/1:/{Table/68/1-Max}] initiating a split of this range at key /Table/68/2 [r71] (zone config) I190903 20:13:00.733530 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:00.758767 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p56 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:00.760143 44693 storage/replica_command.go:284 [n1,split,s1,r50/1:/Table/68/{1-2}] initiating a split of this range at key /Table/68/1/3/4 [r72] (zone config) I190903 20:13:00.761625 44808 storage/replica_command.go:1430 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r71:/{Table/68/2-Max} [(n1,s1):1, next=2, gen=51] I190903 20:13:00.768084 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:00.770023 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.772100 44808 storage/replica_raft.go:291 [n1,s1,r71/1:/{Table/68/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:00.773267 45047 storage/replica_command.go:284 [n1,split,s1,r72/1:/Table/68/{1/3/4-2}] initiating a split of this range at key /Table/68/1/3/5 [r73] (zone config) I190903 20:13:00.774014 44808 storage/store_snapshot.go:995 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] sending LEARNER snapshot 9f13f25c at applied index 14 I190903 20:13:00.774241 44808 storage/store_snapshot.go:1038 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:00.775323 45108 storage/replica_raftstorage.go:793 [n3,s3,r71/2:{-}] applying LEARNER snapshot [id=9f13f25c index=14] I190903 20:13:00.776855 45108 storage/replica_raftstorage.go:814 [n3,s3,r71/2:/{Table/68/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=9f13f25c index=14] I190903 20:13:00.777571 44808 storage/replica_command.go:1430 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r71:/{Table/68/2-Max} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=52] I190903 20:13:00.785365 44808 storage/replica_raft.go:291 [n1,s1,r71/1:/{Table/68/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:00.785929 44808 storage/replica_command.go:1430 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r71:/{Table/68/2-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=53] I190903 20:13:00.785977 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.789975 44896 storage/replica_command.go:284 [n1,split,s1,r73/1:/Table/68/{1/3/5-2}] initiating a split of this range at key /Table/68/1/5/6 [r74] (zone config) I190903 20:13:00.791686 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.794377 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.796143 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.798124 44808 storage/replica_raft.go:291 [n1,s1,r71/1:/{Table/68/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:00.798210 44808 storage/replica_raft.go:304 [n1,s1,r71/1:/{Table/68/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:00.798236 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.799132 44808 storage/replica_command.go:979 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] could not promote [] to voter, rolling back: change replicas of r71 failed: storage/replica_raft.go:305: [n1,s1,r71/1:/{Table/68/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:00.799176 44808 storage/queue.go:1027 [n1,replicate,s1,r71/1:/{Table/68/2-Max}] change replicas of r71 failed: storage/replica_raft.go:305: [n1,s1,r71/1:/{Table/68/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:00.800310 45199 storage/replica_command.go:1430 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r72:/Table/68/1/3/{4-5} [(n1,s1):1, next=2, gen=53] I190903 20:13:00.801005 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.802985 45268 storage/replica_command.go:284 [n1,split,s1,r74/1:/Table/68/{1/5/6-2}] initiating a split of this range at key /Table/68/1/5/7 [r75] (zone config) I190903 20:13:00.803421 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:00.804109 45199 storage/replica_raft.go:291 [n1,s1,r72/1:/Table/68/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:13:00.805257 45199 storage/store_snapshot.go:995 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] sending LEARNER snapshot 53e0ead8 at applied index 17 I190903 20:13:00.805401 45199 storage/store_snapshot.go:1038 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:00.805506 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.807345 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.807892 45300 storage/replica_raftstorage.go:793 [n2,s2,r72/2:{-}] applying LEARNER snapshot [id=53e0ead8 index=17] I190903 20:13:00.809142 45300 storage/replica_raftstorage.go:814 [n2,s2,r72/2:/Table/68/1/3/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=53e0ead8 index=17] I190903 20:13:00.809743 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.809814 45199 storage/replica_command.go:1430 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r72:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=54] I190903 20:13:00.813702 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:00.815002 45199 storage/replica_raft.go:291 [n1,s1,r72/1:/Table/68/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:13:00.815313 45318 storage/replica_command.go:284 [n1,split,s1,r75/1:/Table/68/{1/5/7-2}] initiating a split of this range at key /Table/68/1/5/8 [r76] (zone config) I190903 20:13:00.815522 45199 storage/replica_command.go:1430 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r72:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=55] I190903 20:13:00.817607 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:00.819762 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.821418 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.823409 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:00.824980 45199 storage/replica_raft.go:291 [n1,s1,r72/1:/Table/68/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:00.825047 45199 storage/replica_raft.go:304 [n1,s1,r72/1:/Table/68/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:00.825323 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:00.826119 45199 storage/replica_command.go:979 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r72 failed: storage/replica_raft.go:305: [n1,s1,r72/1:/Table/68/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:00.826149 45199 storage/queue.go:1027 [n1,replicate,s1,r72/1:/Table/68/1/3/{4-5}] change replicas of r72 failed: storage/replica_raft.go:305: [n1,s1,r72/1:/Table/68/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:00.826812 44979 storage/replica_command.go:1430 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r74:/Table/68/1/5/{6-7} [(n1,s1):1, next=2, gen=55] I190903 20:13:00.829086 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.831273 44979 storage/replica_raft.go:291 [n1,s1,r74/1:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:00.832492 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.832570 44979 storage/store_snapshot.go:995 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] sending LEARNER snapshot 547a1e83 at applied index 17 I190903 20:13:00.832772 44979 storage/store_snapshot.go:1038 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:00.834850 45371 storage/replica_raftstorage.go:793 [n3,s3,r74/2:{-}] applying LEARNER snapshot [id=547a1e83 index=17] I190903 20:13:00.836183 45371 storage/replica_raftstorage.go:814 [n3,s3,r74/2:/Table/68/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=547a1e83 index=17] I190903 20:13:00.836223 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.836629 44979 storage/replica_command.go:1430 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] change replicas (add [(n3,s3):2] remove []): existing descriptor r74:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=56] I190903 20:13:00.838361 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.840397 44979 storage/replica_raft.go:291 [n1,s1,r74/1:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:00.840453 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.840889 44979 storage/replica_command.go:1430 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r74:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=57] I190903 20:13:00.847260 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.849266 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.851192 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.851208 44979 storage/replica_raft.go:291 [n1,s1,r74/1:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:00.851352 44979 storage/replica_raft.go:304 [n1,s1,r74/1:/Table/68/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:00.852385 44979 storage/replica_command.go:979 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r74 failed: storage/replica_raft.go:305: [n1,s1,r74/1:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:00.852428 44979 storage/queue.go:1027 [n1,replicate,s1,r74/1:/Table/68/1/5/{6-7}] change replicas of r74 failed: storage/replica_raft.go:305: [n1,s1,r74/1:/Table/68/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:00.853540 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.855450 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.857149 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.859500 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.861856 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:00.866287 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.868231 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:00.870322 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:00.872238 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.874131 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.898824 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.902174 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.908131 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 E190903 20:13:00.912061 45610 storage/queue.go:1027 [n3,replicate,s3,r71/2:/{Table/68/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 20:13:00.932433 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.942809 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.950413 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.952536 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.956558 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.962709 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:00.973219 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:00.975243 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:00.997800 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:01.000994 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:01.002520 45838 storage/replica_command.go:1430 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r55:/Table/6{2/2-4} [(n1,s1):2, next=3, gen=32] I190903 20:13:01.003172 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:01.008119 45838 storage/replica_raft.go:291 [n1,s1,r55/2:/Table/6{2/2-4}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:01.015195 45838 storage/store_snapshot.go:995 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] sending LEARNER snapshot 818c247b at applied index 36 I190903 20:13:01.015371 45838 storage/store_snapshot.go:1038 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:01.016450 45760 storage/replica_raftstorage.go:793 [n2,s2,r55/3:{-}] applying LEARNER snapshot [id=818c247b index=36] I190903 20:13:01.017861 45760 storage/replica_raftstorage.go:814 [n2,s2,r55/3:/Table/6{2/2-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=818c247b index=36] I190903 20:13:01.019828 45838 storage/replica_command.go:1430 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] change replicas (add [(n2,s2):3] remove []): existing descriptor r55:/Table/6{2/2-4} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=33] I190903 20:13:01.027220 45838 storage/replica_raft.go:291 [n1,s1,r55/2:/Table/6{2/2-4}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:01.027691 45838 storage/replica_command.go:1430 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r55:/Table/6{2/2-4} [(n1,s1):2, (n2,s2):3, next=4, gen=34] I190903 20:13:01.039618 45838 storage/replica_raft.go:291 [n1,s1,r55/2:/Table/6{2/2-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:01.039670 45838 storage/replica_raft.go:304 [n1,s1,r55/2:/Table/6{2/2-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:01.041809 45838 storage/replica_command.go:979 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] could not promote [] to voter, rolling back: change replicas of r55 failed: storage/replica_raft.go:305: [n1,s1,r55/2:/Table/6{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:01.041881 45838 storage/queue.go:1027 [n1,replicate,s1,r55/2:/Table/6{2/2-4}] change replicas of r55 failed: storage/replica_raft.go:305: [n1,s1,r55/2:/Table/6{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:01.042197 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:01.045092 45985 storage/replica_command.go:1430 [n1,replicate,s1,r49/1:/Table/68{-/1}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r49:/Table/68{-/1} [(n1,s1):1, next=2, gen=50] I190903 20:13:01.066618 45985 storage/replica_raft.go:291 [n1,s1,r49/1:/Table/68{-/1}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:01.067707 45985 storage/store_snapshot.go:995 [n1,replicate,s1,r49/1:/Table/68{-/1}] sending LEARNER snapshot 4e58bb21 at applied index 18 I190903 20:13:01.067821 46004 storage/replica_command.go:1430 [n3,replicate,s3,r71/2:/{Table/68/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r71:/{Table/68/2-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=53] I190903 20:13:01.067859 45985 storage/store_snapshot.go:1038 [n1,replicate,s1,r49/1:/Table/68{-/1}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:01.069244 46072 storage/replica_raftstorage.go:793 [n3,s3,r49/2:{-}] applying LEARNER snapshot [id=4e58bb21 index=18] I190903 20:13:01.070588 46072 storage/replica_raftstorage.go:814 [n3,s3,r49/2:/Table/68{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=4e58bb21 index=18] I190903 20:13:01.071113 45985 storage/replica_command.go:1430 [n1,replicate,s1,r49/1:/Table/68{-/1}] change replicas (add [(n3,s3):2] remove []): existing descriptor r49:/Table/68{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=51] I190903 20:13:01.074686 45985 storage/replica_raft.go:291 [n1,s1,r49/1:/Table/68{-/1}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:01.075206 45985 storage/replica_command.go:1430 [n1,replicate,s1,r49/1:/Table/68{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r49:/Table/68{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=52] I190903 20:13:01.076168 46004 storage/replica_raft.go:291 [n3,s3,r71/2:/{Table/68/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:01.087187 45985 storage/replica_raft.go:291 [n1,s1,r49/1:/Table/68{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:01.087234 45985 storage/replica_raft.go:304 [n1,s1,r49/1:/Table/68{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:01.088006 45985 storage/replica_command.go:979 [n1,replicate,s1,r49/1:/Table/68{-/1}] could not promote [] to voter, rolling back: change replicas of r49 failed: storage/replica_raft.go:305: [n1,s1,r49/1:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:01.088047 45985 storage/queue.go:1027 [n1,replicate,s1,r49/1:/Table/68{-/1}] change replicas of r49 failed: storage/replica_raft.go:305: [n1,s1,r49/1:/Table/68{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:01.107082 46204 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [(n2,s2):5LEARNER] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, next=5, gen=20] I190903 20:13:01.110731 46204 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing ADD_REPLICA[(n2,s2):5LEARNER]: after=[(n1,s1):3 (n2,s2):5LEARNER] next=6 I190903 20:13:01.111759 46204 storage/store_snapshot.go:995 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] sending LEARNER snapshot 4010bb79 at applied index 66 I190903 20:13:01.111939 46204 storage/store_snapshot.go:1038 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] streamed snapshot to (n2,s2):5LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:01.113016 46209 storage/replica_raftstorage.go:793 [n2,s2,r25/5:{-}] applying LEARNER snapshot [id=4010bb79 index=66] I190903 20:13:01.114687 46209 storage/replica_raftstorage.go:814 [n2,s2,r25/5:/Table/5{6/2-8}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=4010bb79 index=66] I190903 20:13:01.115814 46204 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [(n2,s2):5] remove []): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n2,s2):5LEARNER, next=6, gen=21] I190903 20:13:01.123040 46204 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing ADD_REPLICA[(n2,s2):5]: after=[(n1,s1):3 (n2,s2):5] next=6 I190903 20:13:01.123538 46204 storage/replica_command.go:1430 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n2,s2):5, next=6, gen=22] I190903 20:13:01.136450 46204 storage/replica_raft.go:291 [n1,s1,r25/3:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):5] next=6 E190903 20:13:01.136487 46204 storage/replica_raft.go:304 [n1,s1,r25/3:/Table/5{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):5] next=6 to remove self (leaseholder) I190903 20:13:01.138065 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:01.139640 46204 storage/replica_command.go:979 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] could not promote [] to voter, rolling back: change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/3:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):5] next=6 to remove self (leaseholder) E190903 20:13:01.139677 46204 storage/queue.go:1027 [n1,replicate,s1,r25/3:/Table/5{6/2-8}] change replicas of r25 failed: storage/replica_raft.go:305: [n1,s1,r25/3:/Table/5{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):5] next=6 to remove self (leaseholder) I190903 20:13:01.141152 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:01.143424 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:01.146412 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:01.149022 45921 storage/store.go:2593 [n1,replicaGC,s1,r71/1:/{Table/68/2-Max}] removing replica r71/1 I190903 20:13:01.149326 45921 storage/replica_destroy.go:152 [n1,replicaGC,s1,r71/1:/{Table/68/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:01.149935 46316 storage/queue.go:1027 [n2,replicate,s2,r72/2:/Table/68/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:01.209398 46525 storage/replica_command.go:1430 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):1, next=2, gen=49] I190903 20:13:01.214276 46525 storage/replica_raft.go:291 [n1,s1,r48/1:/Table/6{6/2-8}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:01.215848 46525 storage/store_snapshot.go:995 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] sending LEARNER snapshot c84669ef at applied index 21 I190903 20:13:01.216056 46525 storage/store_snapshot.go:1038 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:01.217982 46548 storage/replica_raftstorage.go:793 [n3,s3,r48/2:{-}] applying LEARNER snapshot [id=c84669ef index=21] I190903 20:13:01.226155 46548 storage/replica_raftstorage.go:814 [n3,s3,r48/2:/Table/6{6/2-8}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=c84669ef index=21] I190903 20:13:01.227111 46525 storage/replica_command.go:1430 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] change replicas (add [(n3,s3):2] remove []): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=50] I190903 20:13:01.232128 46525 storage/replica_raft.go:291 [n1,s1,r48/1:/Table/6{6/2-8}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:01.232665 46525 storage/replica_command.go:1430 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):1, (n3,s3):2, next=3, gen=51] I190903 20:13:01.244309 46525 storage/replica_raft.go:291 [n1,s1,r48/1:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:01.244348 46525 storage/replica_raft.go:304 [n1,s1,r48/1:/Table/6{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:01.245406 46525 storage/replica_command.go:979 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] could not promote [] to voter, rolling back: change replicas of r48 failed: storage/replica_raft.go:305: [n1,s1,r48/1:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:01.245438 46525 storage/queue.go:1027 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] change replicas of r48 failed: storage/replica_raft.go:305: [n1,s1,r48/1:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:01.254992 46608 storage/queue.go:1027 [n3,replicate,s3,r74/2:/Table/68/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:01.282608 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:01.329539 1776 server/status/runtime.go:498 [n2] runtime stats: 223 MiB RSS, 985 goroutines, 52 MiB/49 MiB/138 MiB GO alloc/idle/total, 53 MiB/101 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (31x), 104 MiB/104 MiB (r/w)net E190903 20:13:01.442364 47169 storage/queue.go:1027 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 20:13:01.456715 2354 server/status/runtime.go:498 [n3] runtime stats: 225 MiB RSS, 985 goroutines, 60 MiB/43 MiB/138 MiB GO alloc/idle/total, 53 MiB/101 MiB CGO alloc/total, 0.0 CGO/sec, 0.0/0.0 %(u/s)time, 0.0 %gc (31x), 103 MiB/103 MiB (r/w)net I190903 20:13:01.470135 47226 storage/replica_command.go:1430 [n2,replicate,s2,r72/2:/Table/68/1/3/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r72:/Table/68/1/3/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=55] I190903 20:13:01.478420 47226 storage/replica_raft.go:291 [n2,s2,r72/2:/Table/68/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:13:01.548824 47341 storage/store.go:2593 [n1,replicaGC,s1,r72/1:/Table/68/1/3/{4-5}] removing replica r72/1 I190903 20:13:01.549204 47341 storage/replica_destroy.go:152 [n1,replicaGC,s1,r72/1:/Table/68/1/3/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:01.576187 46690 storage/queue.go:1027 [n3,replicate,s3,r49/2:/Table/68{-/1}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:01.600437 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:01.606409 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:01.610468 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:01.614505 47279 storage/replica_command.go:1430 [n3,replicate,s3,r74/2:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r74:/Table/68/1/5/{6-7} [(n1,s1):1, (n3,s3):2, next=3, gen=57] I190903 20:13:01.640218 47279 storage/replica_raft.go:291 [n3,s3,r74/2:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:01.649699 47608 storage/replica_command.go:1430 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r55:/Table/6{2/2-4} [(n1,s1):2, (n2,s2):3, next=4, gen=34] I190903 20:13:01.650774 47803 storage/store.go:2593 [n1,replicaGC,s1,r74/1:/Table/68/1/5/{6-7}] removing replica r74/1 I190903 20:13:01.651613 47803 storage/replica_destroy.go:152 [n1,replicaGC,s1,r74/1:/Table/68/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:01.657793 47608 storage/replica_raft.go:291 [n2,s2,r55/3:/Table/6{2/2-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:01.662025 46539 storage/replica_command.go:1430 [n2,replicate,s2,r25/5:/Table/5{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r25:/Table/5{6/2-8} [(n1,s1):3, (n2,s2):5, next=6, gen=22] I190903 20:13:01.671001 46539 storage/replica_raft.go:291 [n2,s2,r25/5:/Table/5{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):5] next=6 E190903 20:13:01.671603 47811 storage/queue.go:1027 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] no removable replicas from range that needs a removal: [1:0, 2*:28] I190903 20:13:01.720305 44522 storage/store.go:2593 [n1,replicaGC,s1,r55/2:/Table/6{2/2-4}] removing replica r55/2 I190903 20:13:01.720696 44522 storage/replica_destroy.go:152 [n1,replicaGC,s1,r55/2:/Table/6{2/2-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:01.721003 47997 storage/store.go:2593 [n1,replicaGC,s1,r25/3:/Table/5{6/2-8}] removing replica r25/3 I190903 20:13:01.721398 47997 storage/replica_destroy.go:152 [n1,replicaGC,s1,r25/3:/Table/5{6/2-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:01.910705 48550 storage/replica_command.go:1430 [n3,replicate,s3,r49/2:/Table/68{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r49:/Table/68{-/1} [(n1,s1):1, (n3,s3):2, next=3, gen=52] I190903 20:13:01.919400 48550 storage/replica_raft.go:291 [n3,s3,r49/2:/Table/68{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:01.927592 48615 storage/store.go:2593 [n1,replicaGC,s1,r49/1:/Table/68{-/1}] removing replica r49/1 I190903 20:13:01.927917 48615 storage/replica_destroy.go:152 [n1,replicaGC,s1,r49/1:/Table/68{-/1}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:02.098165 49067 storage/queue.go:1027 [n1,replicate,s1,r48/1:/Table/6{6/2-8}] no removable replicas from range that needs a removal: [1*:32, 2:0] I190903 20:13:02.158192 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:02.159890 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:02.162649 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:02.164221 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:02.166684 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:02.168313 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:02.171109 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=2 mtime=2019-09-03 20:13:02.169991012 +0000 UTC I190903 20:13:02.177036 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 1}] I190903 20:13:02.177428 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:02.177510 49270 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=2:1567541887.490391344,0, refcount=0 I190903 20:13:02.228036 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=68 (multi col list partitioning - DEFAULT) version=3 mtime=2019-09-03 20:13:02.227871185 +0000 UTC I190903 20:13:02.234233 49645 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=3:1567541888.612480369,0, refcount=0 I190903 20:13:02.234284 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 2}] I190903 20:13:02.261876 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=4 mtime=2019-09-03 20:13:02.259815061 +0000 UTC I190903 20:13:02.271427 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 68, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:02.273230 50149 storage/replica_command.go:1430 [n3,s3,r74/2:/Table/68/1/5/{6-7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r74:/Table/68/1/5/{6-7} [(n3,s3):2, next=3, gen=58] I190903 20:13:02.274042 49651 sql/lease.go:923 new lease: 68("multi col list partitioning - DEFAULT") ver=4:1567541889.850064922,0, refcount=0 I190903 20:13:02.274671 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 3}] I190903 20:13:02.283020 50149 storage/replica_raft.go:291 [n3,s3,r74/2:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:02.283885 50149 storage/store_snapshot.go:995 [n3,s3,r74/2:/Table/68/1/5/{6-7}] sending LEARNER snapshot 391deb13 at applied index 34 I190903 20:13:02.284017 50149 storage/store_snapshot.go:1038 [n3,s3,r74/2:/Table/68/1/5/{6-7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:02.285142 50162 storage/replica_raftstorage.go:793 [n1,s1,r74/3:{-}] applying LEARNER snapshot [id=391deb13 index=34] I190903 20:13:02.291926 50162 storage/replica_raftstorage.go:814 [n1,s1,r74/3:/Table/68/1/5/{6-7}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=391deb13 index=34] E190903 20:13:02.292023 50289 storage/queue.go:1027 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] no removable replicas from range that needs a removal: [1:0, 2*:35] I190903 20:13:02.292403 50149 storage/replica_command.go:1430 [n3,s3,r74/2:/Table/68/1/5/{6-7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r74:/Table/68/1/5/{6-7} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=59] I190903 20:13:02.299130 50149 storage/replica_raft.go:291 [n3,s3,r74/2:/Table/68/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 E190903 20:13:02.301610 50304 storage/queue.go:1027 [n1,replicate,s1,r74/3:/Table/68/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:38] I190903 20:13:02.313164 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:02.323220 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:02.331732 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:02.362570 51040 storage/replica_command.go:1430 [n1,s1,r74/3:/Table/68/1/5/{6-7}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r74:/Table/68/1/5/{6-7} [(n3,s3):2, (n1,s1):3, next=4, gen=60] I190903 20:13:02.371625 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:02.372465 51116 storage/replica_command.go:284 [n1,split,s1,r76/1:/Table/68/{1/5/8-2}] initiating a split of this range at key /Table/68/1/5/9 [r77] (zone config) I190903 20:13:02.372795 51040 storage/replica_raft.go:291 [n1,s1,r74/3:/Table/68/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:02.378918 50052 storage/replica_command.go:609 [n1,merge,s1,r73/1:/Table/68/1/{3/5-5/6}] initiating a merge of r74:/Table/68/1/5/{6-7} [(n1,s1):3, next=4, gen=61] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:02.386528 51325 storage/store.go:2593 [n3,replicaGC,s3,r74/2:/Table/68/1/5/{6-7}] removing replica r74/2 I190903 20:13:02.386908 51325 storage/replica_destroy.go:152 [n3,replicaGC,s3,r74/2:/Table/68/1/5/{6-7}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:02.388625 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 68, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:02.388750 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.388972 193 storage/store.go:2593 [n1,s1,r73/1:/Table/68/1/{3/5-5/6}] removing replica r74/3 I190903 20:13:02.391458 51429 storage/replica_command.go:284 [n1,split,s1,r77/1:/Table/68/{1/5/9-2}] initiating a split of this range at key /Table/68/1/6 [r78] (zone config) I190903 20:13:02.391759 51590 storage/replica_command.go:284 [n1,split,s1,r73/1:/Table/68/1/{3/5-5/7}] initiating a split of this range at key /Table/68/1/5 [r79] (zone config) I190903 20:13:02.400579 51435 storage/replica_command.go:1430 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r77:/Table/68/1/{5/9-6} [(n1,s1):1, next=2, gen=58] I190903 20:13:02.419824 51435 storage/replica_raft.go:291 [n1,s1,r77/1:/Table/68/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:13:02.420980 51435 storage/store_snapshot.go:995 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] sending LEARNER snapshot 8ddec0e1 at applied index 17 I190903 20:13:02.422078 51435 storage/store_snapshot.go:1038 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:02.424759 51784 storage/replica_raftstorage.go:793 [n2,s2,r77/2:{-}] applying LEARNER snapshot [id=8ddec0e1 index=17] I190903 20:13:02.428582 51784 storage/replica_raftstorage.go:814 [n2,s2,r77/2:/Table/68/1/{5/9-6}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=8ddec0e1 index=17] I190903 20:13:02.430106 51435 storage/replica_command.go:1430 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r77:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=59] I190903 20:13:02.434210 51435 storage/replica_raft.go:291 [n1,s1,r77/1:/Table/68/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:13:02.434702 51435 storage/replica_command.go:1430 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r77:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=60] I190903 20:13:02.436511 51091 storage/replica_command.go:1430 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):1, (n3,s3):2, next=3, gen=51] I190903 20:13:02.445077 51435 storage/replica_raft.go:291 [n1,s1,r77/1:/Table/68/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:02.445118 51435 storage/replica_raft.go:304 [n1,s1,r77/1:/Table/68/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:02.446276 51435 storage/replica_command.go:979 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r77 failed: storage/replica_raft.go:305: [n1,s1,r77/1:/Table/68/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:02.446343 51435 storage/queue.go:1027 [n1,replicate,s1,r77/1:/Table/68/1/{5/9-6}] change replicas of r77 failed: storage/replica_raft.go:305: [n1,s1,r77/1:/Table/68/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:02.446596 51091 storage/replica_raft.go:291 [n3,s3,r48/2:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:02.446765 51729 storage/replica_command.go:1430 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r75:/Table/68/1/5/{7-8} [(n1,s1):1, next=2, gen=56] I190903 20:13:02.449499 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:02.449815 51729 storage/replica_raft.go:291 [n1,s1,r75/1:/Table/68/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:02.450764 51729 storage/store_snapshot.go:995 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] sending LEARNER snapshot f9ee7fc6 at applied index 20 I190903 20:13:02.450909 51729 storage/store_snapshot.go:1038 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:02.452423 51885 storage/replica_raftstorage.go:793 [n3,s3,r75/2:{-}] applying LEARNER snapshot [id=f9ee7fc6 index=20] I190903 20:13:02.454018 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.456668 51417 storage/store.go:2593 [n1,replicaGC,s1,r48/1:/Table/6{6/2-8}] removing replica r48/1 I190903 20:13:02.456993 51417 storage/replica_destroy.go:152 [n1,replicaGC,s1,r48/1:/Table/6{6/2-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:02.458600 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.460664 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:02.463296 51885 storage/replica_raftstorage.go:814 [n3,s3,r75/2:/Table/68/1/5/{7-8}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=f9ee7fc6 index=20] I190903 20:13:02.463822 51729 storage/replica_command.go:1430 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] change replicas (add [(n3,s3):2] remove []): existing descriptor r75:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=57] I190903 20:13:02.465435 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.467105 51729 storage/replica_raft.go:291 [n1,s1,r75/1:/Table/68/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:02.467617 51729 storage/replica_command.go:1430 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r75:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=58] I190903 20:13:02.468243 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.470221 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:02.472548 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:02.477226 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.478051 51729 storage/replica_raft.go:291 [n1,s1,r75/1:/Table/68/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:02.478124 51729 storage/replica_raft.go:304 [n1,s1,r75/1:/Table/68/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:02.479539 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.479787 51729 storage/replica_command.go:979 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r75 failed: storage/replica_raft.go:305: [n1,s1,r75/1:/Table/68/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:02.479828 51729 storage/queue.go:1027 [n1,replicate,s1,r75/1:/Table/68/1/5/{7-8}] change replicas of r75 failed: storage/replica_raft.go:305: [n1,s1,r75/1:/Table/68/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:02.480236 51980 storage/replica_command.go:1430 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r79:/Table/68/1/5{-/7} [(n1,s1):1, next=2, gen=63] I190903 20:13:02.481725 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.484087 51980 storage/replica_raft.go:291 [n1,s1,r79/1:/Table/68/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:13:02.485151 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.485468 51980 storage/store_snapshot.go:995 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] sending LEARNER snapshot 2f1cb1e8 at applied index 14 I190903 20:13:02.485700 51980 storage/store_snapshot.go:1038 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 20, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:02.486995 52037 storage/replica_raftstorage.go:793 [n2,s2,r79/2:{-}] applying LEARNER snapshot [id=2f1cb1e8 index=14] I190903 20:13:02.488155 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:02.488449 52037 storage/replica_raftstorage.go:814 [n2,s2,r79/2:/Table/68/1/5{-/7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=2f1cb1e8 index=14] I190903 20:13:02.489141 51980 storage/replica_command.go:1430 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r79:/Table/68/1/5{-/7} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=64] I190903 20:13:02.490605 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.493288 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.494039 51980 storage/replica_raft.go:291 [n1,s1,r79/1:/Table/68/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:13:02.494528 51980 storage/replica_command.go:1430 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r79:/Table/68/1/5{-/7} [(n1,s1):1, (n2,s2):2, next=3, gen=65] I190903 20:13:02.495473 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.497442 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.499527 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.501310 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.503480 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:02.504540 51980 storage/replica_raft.go:291 [n1,s1,r79/1:/Table/68/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:02.504574 51980 storage/replica_raft.go:304 [n1,s1,r79/1:/Table/68/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:02.505725 51980 storage/replica_command.go:979 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r79 failed: storage/replica_raft.go:305: [n1,s1,r79/1:/Table/68/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:02.505774 51980 storage/queue.go:1027 [n1,replicate,s1,r79/1:/Table/68/1/5{-/7}] change replicas of r79 failed: storage/replica_raft.go:305: [n1,s1,r79/1:/Table/68/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:02.506253 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 E190903 20:13:02.507911 52166 storage/queue.go:1027 [n3,replicate,s3,r75/2:/Table/68/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:26] E190903 20:13:02.509020 52181 storage/queue.go:1027 [n2,replicate,s2,r79/2:/Table/68/1/5{-/7}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 20:13:02.509815 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.511845 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.559176 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:02.561736 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.563366 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.567338 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:02.569854 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:02.571216 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.572606 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.574397 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.577200 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 E190903 20:13:02.599049 52468 storage/queue.go:1027 [n2,replicate,s2,r77/2:/Table/68/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:02.643175 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:02.690445 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:02.692881 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:02.696205 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.698443 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.702123 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:02.704962 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:02.707550 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.710910 52817 storage/replica_command.go:1430 [n2,replicate,s2,r79/2:/Table/68/1/5{-/7}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r79:/Table/68/1/5{-/7} [(n1,s1):1, (n2,s2):2, next=3, gen=65] I190903 20:13:02.711445 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.714103 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:02.717309 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:02.723162 52817 storage/replica_raft.go:291 [n2,s2,r79/2:/Table/68/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:13:02.738274 52370 storage/store.go:2593 [n1,replicaGC,s1,r79/1:/Table/68/1/5{-/7}] removing replica r79/1 I190903 20:13:02.738742 52370 storage/replica_destroy.go:152 [n1,replicaGC,s1,r79/1:/Table/68/1/5{-/7}] removed 11 (0+11) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:02.740255 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:02.746032 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:02.751388 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:02.753118 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:02.754807 42962 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT/multi_col_list_partitioning I190903 20:13:02.776144 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=68 (multi col list partitioning - DEFAULT) version=5 mtime=2019-09-03 20:13:02.764207362 +0000 UTC I190903 20:13:02.785005 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 67, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT"]} I190903 20:13:02.785065 53011 storage/replica_command.go:1430 [n3,replicate,s3,r75/2:/Table/68/1/5/{7-8}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r75:/Table/68/1/5/{7-8} [(n1,s1):1, (n3,s3):2, next=3, gen=58] I190903 20:13:02.787092 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 68 4}] I190903 20:13:02.797781 53011 storage/replica_raft.go:291 [n3,s3,r75/2:/Table/68/1/5/{7-8}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:02.806985 53381 storage/store.go:2593 [n1,replicaGC,s1,r75/1:/Table/68/1/5/{7-8}] removing replica r75/1 I190903 20:13:02.807314 53381 storage/replica_destroy.go:152 [n1,replicaGC,s1,r75/1:/Table/68/1/5/{7-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:02.844458 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=68 (multi col list partitioning - DEFAULT) version=6 mtime=2019-09-03 20:13:02.844287787 +0000 UTC I190903 20:13:02.876209 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 69, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:02.890042 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 70, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root} I190903 20:13:02.891746 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:02.891829 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:02.893970 54209 storage/replica_command.go:284 [n3,split,s3,r71/2:/{Table/68/2-Max}] initiating a split of this range at key /Table/70 [r87] (zone config) I190903 20:13:02.898514 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:02.901437 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:02.911370 54523 storage/replica_command.go:284 [n3,split,s3,r87/1:/{Table/70-Max}] initiating a split of this range at key /Table/70/1 [r88] (zone config) I190903 20:13:02.914550 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:02.929913 54705 storage/replica_command.go:284 [n3,split,s3,r88/1:/{Table/70/1-Max}] initiating a split of this range at key /Table/70/1/3/4 [r89] (zone config) I190903 20:13:02.943799 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:02.953029 54763 storage/replica_command.go:284 [n3,split,s3,r89/1:/{Table/70/1/3…-Max}] initiating a split of this range at key /Table/70/1/3/5 [r90] (zone config) I190903 20:13:02.953612 55140 storage/replica_command.go:1430 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r88:/Table/70/1{-/3/4} [(n3,s3):1, next=2, gen=57] I190903 20:13:02.954219 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:02.965843 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:02.966051 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:02.967565 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:02.969271 55140 storage/replica_raft.go:291 [n3,s3,r88/1:/Table/70/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:02.994034 55140 storage/store_snapshot.go:995 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] sending LEARNER snapshot c2576aee at applied index 17 I190903 20:13:02.994228 55140 storage/store_snapshot.go:1038 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 20:13:02.995408 55686 storage/replica_raftstorage.go:793 [n1,s1,r88/2:{-}] applying LEARNER snapshot [id=c2576aee index=17] I190903 20:13:02.995882 55376 storage/replica_command.go:284 [n3,split,s3,r90/1:/{Table/70/1/3…-Max}] initiating a split of this range at key /Table/70/1/5 [r91] (zone config) I190903 20:13:03.003469 55686 storage/replica_raftstorage.go:814 [n1,s1,r88/2:/Table/70/1{-/3/4}] applied LEARNER snapshot [total=8ms ingestion=4@8ms id=c2576aee index=17] I190903 20:13:03.005295 55140 storage/replica_command.go:1430 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r88:/Table/70/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=58] I190903 20:13:03.010139 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:03.015035 55714 storage/replica_command.go:284 [n3,split,s3,r91/1:/{Table/70/1/5-Max}] initiating a split of this range at key /Table/70/1/5/7 [r92] (zone config) I190903 20:13:03.015193 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.017638 55140 storage/replica_raft.go:291 [n3,s3,r88/1:/Table/70/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:03.018190 55140 storage/replica_command.go:1430 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r88:/Table/70/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=59] I190903 20:13:03.018841 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:03.022299 55925 storage/replica_command.go:1430 [n2,replicate,s2,r77/2:/Table/68/1/{5/9-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r77:/Table/68/1/{5/9-6} [(n1,s1):1, (n2,s2):2, next=3, gen=60] I190903 20:13:03.025642 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:03.030915 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.035245 55925 storage/replica_raft.go:291 [n2,s2,r77/2:/Table/68/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:13:03.036271 55140 storage/replica_raft.go:291 [n3,s3,r88/1:/Table/70/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:03.036317 55140 storage/replica_raft.go:304 [n3,s3,r88/1:/Table/70/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.037129 55140 storage/replica_command.go:979 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r88 failed: storage/replica_raft.go:305: [n3,s3,r88/1:/Table/70/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.037173 55140 storage/queue.go:1027 [n3,replicate,s3,r88/1:/Table/70/1{-/3/4}] change replicas of r88 failed: storage/replica_raft.go:305: [n3,s3,r88/1:/Table/70/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.037613 55960 storage/replica_command.go:1430 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r89:/Table/70/1/3/{4-5} [(n3,s3):1, next=2, gen=58] I190903 20:13:03.038932 56036 storage/replica_command.go:284 [n3,split,s3,r92/1:/{Table/70/1/5…-Max}] initiating a split of this range at key /Table/70/1/5/8 [r93] (zone config) I190903 20:13:03.045836 55948 storage/store.go:2593 [n1,replicaGC,s1,r77/1:/Table/68/1/{5/9-6}] removing replica r77/1 I190903 20:13:03.046190 55948 storage/replica_destroy.go:152 [n1,replicaGC,s1,r77/1:/Table/68/1/{5/9-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:03.046348 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:03.049098 55960 storage/replica_raft.go:291 [n3,s3,r89/1:/Table/70/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:03.052582 55960 storage/store_snapshot.go:995 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] sending LEARNER snapshot 301b9dec at applied index 17 I190903 20:13:03.052798 55960 storage/store_snapshot.go:1038 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.054040 56072 storage/replica_raftstorage.go:793 [n2,s2,r89/2:{-}] applying LEARNER snapshot [id=301b9dec index=17] I190903 20:13:03.055417 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.056170 55968 storage/replica_command.go:284 [n3,split,s3,r93/1:/{Table/70/1/5…-Max}] initiating a split of this range at key /Table/70/1/5/9 [r94] (zone config) I190903 20:13:03.061561 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:03.065858 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.070058 56072 storage/replica_raftstorage.go:814 [n2,s2,r89/2:/Table/70/1/3/{4-5}] applied LEARNER snapshot [total=16ms ingestion=4@16ms id=301b9dec index=17] I190903 20:13:03.071071 56251 storage/replica_command.go:284 [n3,split,s3,r94/1:/{Table/70/1/5…-Max}] initiating a split of this range at key /Table/70/1/6 [r95] (zone config) I190903 20:13:03.071562 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:03.072110 55960 storage/replica_command.go:1430 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r89:/Table/70/1/3/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=59] I190903 20:13:03.089666 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:03.093539 55960 storage/replica_raft.go:291 [n3,s3,r89/1:/Table/70/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:03.094072 55960 storage/replica_command.go:1430 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r89:/Table/70/1/3/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=60] I190903 20:13:03.095155 56178 storage/replica_command.go:284 [n3,split,s3,r95/1:/{Table/70/1/6-Max}] initiating a split of this range at key /Table/70/2 [r96] (zone config) I190903 20:13:03.095342 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:03.108071 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.112559 55960 storage/replica_raft.go:291 [n3,s3,r89/1:/Table/70/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:03.112619 55960 storage/replica_raft.go:304 [n3,s3,r89/1:/Table/70/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.114050 55960 storage/replica_command.go:979 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r89 failed: storage/replica_raft.go:305: [n3,s3,r89/1:/Table/70/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:03.114915 55960 storage/queue.go:1027 [n3,replicate,s3,r89/1:/Table/70/1/3/{4-5}] change replicas of r89 failed: storage/replica_raft.go:305: [n3,s3,r89/1:/Table/70/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.114959 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.116718 56283 storage/replica_command.go:1430 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r90:/Table/70/1/{3/5-5} [(n3,s3):1, next=2, gen=59] I190903 20:13:03.124117 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.139688 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:03.144013 56283 storage/replica_raft.go:291 [n3,s3,r90/1:/Table/70/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:03.146001 56283 storage/store_snapshot.go:995 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] sending LEARNER snapshot 0f1ccf9f at applied index 18 I190903 20:13:03.146161 56283 storage/store_snapshot.go:1038 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.147244 56127 storage/replica_raftstorage.go:793 [n1,s1,r90/2:{-}] applying LEARNER snapshot [id=0f1ccf9f index=18] I190903 20:13:03.148442 56127 storage/replica_raftstorage.go:814 [n1,s1,r90/2:/Table/70/1/{3/5-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=0f1ccf9f index=18] I190903 20:13:03.148891 56283 storage/replica_command.go:1430 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] change replicas (add [(n1,s1):2] remove []): existing descriptor r90:/Table/70/1/{3/5-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=60] I190903 20:13:03.156015 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.174425 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:03.179228 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.180264 56283 storage/replica_raft.go:291 [n3,s3,r90/1:/Table/70/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:03.183296 56283 storage/replica_command.go:1430 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r90:/Table/70/1/{3/5-5} [(n3,s3):1, (n1,s1):2, next=3, gen=61] I190903 20:13:03.195566 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.199042 56283 storage/replica_raft.go:291 [n3,s3,r90/1:/Table/70/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:03.200016 56283 storage/replica_raft.go:304 [n3,s3,r90/1:/Table/70/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.200760 56283 storage/replica_command.go:979 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r90 failed: storage/replica_raft.go:305: [n3,s3,r90/1:/Table/70/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.200931 56283 storage/queue.go:1027 [n3,replicate,s3,r90/1:/Table/70/1/{3/5-5}] change replicas of r90 failed: storage/replica_raft.go:305: [n3,s3,r90/1:/Table/70/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.201406 56222 storage/replica_command.go:1430 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r91:/Table/70/1/5{-/7} [(n3,s3):1, next=2, gen=60] I190903 20:13:03.201703 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:03.208668 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:03.216687 56222 storage/replica_raft.go:291 [n3,s3,r91/1:/Table/70/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:03.217731 56222 storage/store_snapshot.go:995 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] sending LEARNER snapshot 1e54872f at applied index 18 I190903 20:13:03.217881 56222 storage/store_snapshot.go:1038 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.219066 56863 storage/replica_raftstorage.go:793 [n2,s2,r91/2:{-}] applying LEARNER snapshot [id=1e54872f index=18] I190903 20:13:03.219569 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.220365 56863 storage/replica_raftstorage.go:814 [n2,s2,r91/2:/Table/70/1/5{-/7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=1e54872f index=18] I190903 20:13:03.220988 56222 storage/replica_command.go:1430 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r91:/Table/70/1/5{-/7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=61] I190903 20:13:03.227720 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:03.229539 56222 storage/replica_raft.go:291 [n3,s3,r91/1:/Table/70/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:03.230067 56222 storage/replica_command.go:1430 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r91:/Table/70/1/5{-/7} [(n3,s3):1, (n2,s2):2, next=3, gen=62] I190903 20:13:03.241559 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:03.260444 56222 storage/replica_raft.go:291 [n3,s3,r91/1:/Table/70/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:03.260546 56222 storage/replica_raft.go:304 [n3,s3,r91/1:/Table/70/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.261830 56222 storage/replica_command.go:979 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r91 failed: storage/replica_raft.go:305: [n3,s3,r91/1:/Table/70/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:03.261869 56222 storage/queue.go:1027 [n3,replicate,s3,r91/1:/Table/70/1/5{-/7}] change replicas of r91 failed: storage/replica_raft.go:305: [n3,s3,r91/1:/Table/70/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.262287 57004 storage/replica_command.go:1430 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r93:/Table/70/1/5/{8-9} [(n3,s3):1, next=2, gen=62] I190903 20:13:03.264791 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:03.270846 57004 storage/replica_raft.go:291 [n3,s3,r93/1:/Table/70/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:03.272369 57004 storage/store_snapshot.go:995 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] sending LEARNER snapshot 5b974c7e at applied index 18 I190903 20:13:03.273011 57004 storage/store_snapshot.go:1038 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.274150 57067 storage/replica_raftstorage.go:793 [n1,s1,r93/2:{-}] applying LEARNER snapshot [id=5b974c7e index=18] I190903 20:13:03.275944 57067 storage/replica_raftstorage.go:814 [n1,s1,r93/2:/Table/70/1/5/{8-9}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=5b974c7e index=18] I190903 20:13:03.276520 57004 storage/replica_command.go:1430 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] change replicas (add [(n1,s1):2] remove []): existing descriptor r93:/Table/70/1/5/{8-9} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=63] I190903 20:13:03.298380 57004 storage/replica_raft.go:291 [n3,s3,r93/1:/Table/70/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:03.298949 57004 storage/replica_command.go:1430 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r93:/Table/70/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=64] I190903 20:13:03.302299 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.323121 57004 storage/replica_raft.go:291 [n3,s3,r93/1:/Table/70/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:03.323166 57004 storage/replica_raft.go:304 [n3,s3,r93/1:/Table/70/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.324001 57004 storage/replica_command.go:979 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r93 failed: storage/replica_raft.go:305: [n3,s3,r93/1:/Table/70/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.324040 57004 storage/queue.go:1027 [n3,replicate,s3,r93/1:/Table/70/1/5/{8-9}] change replicas of r93 failed: storage/replica_raft.go:305: [n3,s3,r93/1:/Table/70/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.324442 56931 storage/replica_command.go:1430 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r94:/Table/70/1/{5/9-6} [(n3,s3):1, next=2, gen=63] I190903 20:13:03.398130 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:03.400553 56931 storage/replica_raft.go:291 [n3,s3,r94/1:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:03.401849 56931 storage/store_snapshot.go:995 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] sending LEARNER snapshot f04f147a at applied index 18 I190903 20:13:03.402021 56931 storage/store_snapshot.go:1038 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.403247 57276 storage/replica_raftstorage.go:793 [n2,s2,r94/2:{-}] applying LEARNER snapshot [id=f04f147a index=18] I190903 20:13:03.406389 57276 storage/replica_raftstorage.go:814 [n2,s2,r94/2:/Table/70/1/{5/9-6}] applied LEARNER snapshot [total=3ms ingestion=4@3ms id=f04f147a index=18] I190903 20:13:03.410119 56931 storage/replica_command.go:1430 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r94:/Table/70/1/{5/9-6} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=64] I190903 20:13:03.421110 56931 storage/replica_raft.go:291 [n3,s3,r94/1:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:03.421618 56931 storage/replica_command.go:1430 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r94:/Table/70/1/{5/9-6} [(n3,s3):1, (n2,s2):2, next=3, gen=65] I190903 20:13:03.437750 56931 storage/replica_raft.go:291 [n3,s3,r94/1:/Table/70/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:03.437798 56931 storage/replica_raft.go:304 [n3,s3,r94/1:/Table/70/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.440016 56931 storage/replica_command.go:979 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r94 failed: storage/replica_raft.go:305: [n3,s3,r94/1:/Table/70/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:03.440057 56931 storage/queue.go:1027 [n3,replicate,s3,r94/1:/Table/70/1/{5/9-6}] change replicas of r94 failed: storage/replica_raft.go:305: [n3,s3,r94/1:/Table/70/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:03.448031 57537 storage/replica_command.go:1430 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r48:/Table/6{6/2-8} [(n3,s3):2, next=3, gen=52] E190903 20:13:03.449989 57288 storage/queue.go:1027 [n1,replicate,s1,r93/2:/Table/70/1/5/{8-9}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:13:03.477857 57537 storage/replica_raft.go:291 [n3,s3,r48/2:/Table/6{6/2-8}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:03.480914 57537 storage/store_snapshot.go:995 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] sending LEARNER snapshot ac9998b0 at applied index 42 I190903 20:13:03.481095 57537 storage/store_snapshot.go:1038 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.482417 57442 storage/replica_raftstorage.go:793 [n1,s1,r48/3:{-}] applying LEARNER snapshot [id=ac9998b0 index=42] I190903 20:13:03.488985 57442 storage/replica_raftstorage.go:814 [n1,s1,r48/3:/Table/6{6/2-8}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=ac9998b0 index=42] I190903 20:13:03.498372 57537 storage/replica_command.go:1430 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] change replicas (add [(n1,s1):3] remove []): existing descriptor r48:/Table/6{6/2-8} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=53] I190903 20:13:03.507220 57537 storage/replica_raft.go:291 [n3,s3,r48/2:/Table/6{6/2-8}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:03.507774 57537 storage/replica_command.go:1430 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r48:/Table/6{6/2-8} [(n3,s3):2, (n1,s1):3, next=4, gen=54] I190903 20:13:03.532805 57537 storage/replica_raft.go:291 [n3,s3,r48/2:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:03.532851 57537 storage/replica_raft.go:304 [n3,s3,r48/2:/Table/6{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:03.533747 57537 storage/replica_command.go:979 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] could not promote [] to voter, rolling back: change replicas of r48 failed: storage/replica_raft.go:305: [n3,s3,r48/2:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:03.533859 57537 storage/queue.go:1027 [n3,replicate,s3,r48/2:/Table/6{6/2-8}] change replicas of r48 failed: storage/replica_raft.go:305: [n3,s3,r48/2:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:03.535492 57942 storage/replica_command.go:1430 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r96:/{Table/70/2-Max} [(n3,s3):1, next=2, gen=64] E190903 20:13:03.536877 57803 storage/queue.go:1027 [n1,replicate,s1,r88/2:/Table/70/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:13:03.537251 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:03.545894 203 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:03.559994 57942 storage/replica_raft.go:291 [n3,s3,r96/1:/{Table/70/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:03.568421 58043 storage/raft_snapshot_queue.go:120 [n3,raftsnapshot,s3,r96/1:/{Table/70/2-Max}] skipping snapshot; replica is likely a learner in the process of being added: (n1,s1):2LEARNER I190903 20:13:03.569879 57942 storage/store_snapshot.go:995 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] sending LEARNER snapshot 4cc081a2 at applied index 16 I190903 20:13:03.574697 57942 storage/store_snapshot.go:1038 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:03.575399 57888 storage/replica_raftstorage.go:793 [n1,s1,r96/2:{-}] applying LEARNER snapshot [id=4cc081a2 index=16] I190903 20:13:03.578101 57888 storage/replica_raftstorage.go:814 [n1,s1,r96/2:/{Table/70/2-Max}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=4cc081a2 index=16] I190903 20:13:03.580341 57942 storage/replica_command.go:1430 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r96:/{Table/70/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=65] I190903 20:13:03.588239 57942 storage/replica_raft.go:291 [n3,s3,r96/1:/{Table/70/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:03.588760 57942 storage/replica_command.go:1430 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r96:/{Table/70/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=66] I190903 20:13:03.596953 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:03.601772 57942 storage/replica_raft.go:291 [n3,s3,r96/1:/{Table/70/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:03.601819 57942 storage/replica_raft.go:304 [n3,s3,r96/1:/{Table/70/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.602992 57942 storage/replica_command.go:979 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] could not promote [] to voter, rolling back: change replicas of r96 failed: storage/replica_raft.go:305: [n3,s3,r96/1:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.603563 57942 storage/queue.go:1027 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] change replicas of r96 failed: storage/replica_raft.go:305: [n3,s3,r96/1:/{Table/70/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.606324 58145 storage/queue.go:1027 [n1,replicate,s1,r96/2:/{Table/70/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 20:13:03.606371 58030 storage/replica_command.go:1430 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r95:/Table/70/{1/6-2} [(n3,s3):1, next=2, gen=64] I190903 20:13:03.613805 58030 storage/replica_raft.go:291 [n3,s3,r95/1:/Table/70/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:03.616127 58030 storage/store_snapshot.go:995 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] sending LEARNER snapshot 1ea8ad90 at applied index 21 I190903 20:13:03.616290 58030 storage/store_snapshot.go:1038 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.617407 57682 storage/replica_raftstorage.go:793 [n1,s1,r95/2:{-}] applying LEARNER snapshot [id=1ea8ad90 index=21] I190903 20:13:03.629287 57682 storage/replica_raftstorage.go:814 [n1,s1,r95/2:/Table/70/{1/6-2}] applied LEARNER snapshot [total=12ms ingestion=4@1ms id=1ea8ad90 index=21] I190903 20:13:03.629945 58030 storage/replica_command.go:1430 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r95:/Table/70/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=65] I190903 20:13:03.639630 58030 storage/replica_raft.go:291 [n3,s3,r95/1:/Table/70/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:03.640190 58030 storage/replica_command.go:1430 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r95:/Table/70/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=66] I190903 20:13:03.658257 58030 storage/replica_raft.go:291 [n3,s3,r95/1:/Table/70/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:03.658303 58030 storage/replica_raft.go:304 [n3,s3,r95/1:/Table/70/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.659118 58030 storage/replica_command.go:979 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r95 failed: storage/replica_raft.go:305: [n3,s3,r95/1:/Table/70/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:03.659144 58030 storage/queue.go:1027 [n3,replicate,s3,r95/1:/Table/70/{1/6-2}] change replicas of r95 failed: storage/replica_raft.go:305: [n3,s3,r95/1:/Table/70/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:03.659573 58398 storage/replica_command.go:1430 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r29:/Table/58{-/1} [(n3,s3):2, next=3, gen=16] I190903 20:13:03.672643 58398 storage/replica_raft.go:291 [n3,s3,r29/2:/Table/58{-/1}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:03.673851 58398 storage/store_snapshot.go:995 [n3,replicate,s3,r29/2:/Table/58{-/1}] sending LEARNER snapshot 7ab6ae65 at applied index 40 I190903 20:13:03.674019 58398 storage/store_snapshot.go:1038 [n3,replicate,s3,r29/2:/Table/58{-/1}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:03.675343 58354 storage/replica_raftstorage.go:793 [n1,s1,r29/3:{-}] applying LEARNER snapshot [id=7ab6ae65 index=40] I190903 20:13:03.677568 58493 storage/replica_command.go:1430 [n1,replicate,s1,r88/2:/Table/70/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r88:/Table/70/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=59] I190903 20:13:03.679888 58354 storage/replica_raftstorage.go:814 [n1,s1,r29/3:/Table/58{-/1}] applied LEARNER snapshot [total=4ms ingestion=4@4ms id=7ab6ae65 index=40] I190903 20:13:03.684097 58398 storage/replica_command.go:1430 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas (add [(n1,s1):3] remove []): existing descriptor r29:/Table/58{-/1} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=17] I190903 20:13:03.691243 58493 storage/replica_raft.go:291 [n1,s1,r88/2:/Table/70/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:03.704974 58559 storage/store.go:2593 [n3,replicaGC,s3,r88/1:/Table/70/1{-/3/4}] removing replica r88/1 I190903 20:13:03.705334 58559 storage/replica_destroy.go:152 [n3,replicaGC,s3,r88/1:/Table/70/1{-/3/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:03.705397 58398 storage/replica_raft.go:291 [n3,s3,r29/2:/Table/58{-/1}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:03.705932 58398 storage/replica_command.go:1430 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r29:/Table/58{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=18] I190903 20:13:03.725451 58398 storage/replica_raft.go:291 [n3,s3,r29/2:/Table/58{-/1}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:03.725538 58398 storage/replica_raft.go:304 [n3,s3,r29/2:/Table/58{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:03.726424 58398 storage/replica_command.go:979 [n3,replicate,s3,r29/2:/Table/58{-/1}] could not promote [] to voter, rolling back: change replicas of r29 failed: storage/replica_raft.go:305: [n3,s3,r29/2:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:03.726457 58398 storage/queue.go:1027 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas of r29 failed: storage/replica_raft.go:305: [n3,s3,r29/2:/Table/58{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:03.729559 58702 storage/queue.go:1027 [n1,replicate,s1,r90/2:/Table/70/1/{3/5-5}] no removable replicas from range that needs a removal: [1:0, 2*:27] E190903 20:13:03.738425 58728 storage/queue.go:1027 [n2,replicate,s2,r91/2:/Table/70/1/5{-/7}] no removable replicas from range that needs a removal: [1:0, 2*:26] E190903 20:13:03.750613 58733 storage/queue.go:1027 [n2,replicate,s2,r94/2:/Table/70/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:03.758456 58811 storage/queue.go:1027 [n2,replicate,s2,r89/2:/Table/70/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] E190903 20:13:03.809290 58924 storage/queue.go:1027 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] no removable replicas from range that needs a removal: [2:0, 3*:49] I190903 20:13:03.845962 58956 storage/replica_command.go:1430 [n1,replicate,s1,r93/2:/Table/70/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r93:/Table/70/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=64] I190903 20:13:03.849104 58956 storage/replica_raft.go:291 [n1,s1,r93/2:/Table/70/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:03.857021 58958 storage/store.go:2593 [n3,replicaGC,s3,r93/1:/Table/70/1/5/{8-9}] removing replica r93/1 I190903 20:13:03.857443 58958 storage/replica_destroy.go:152 [n3,replicaGC,s3,r93/1:/Table/70/1/5/{8-9}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:03.871713 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:03.911081 59122 storage/replica_command.go:1430 [n1,replicate,s1,r90/2:/Table/70/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r90:/Table/70/1/{3/5-5} [(n3,s3):1, (n1,s1):2, next=3, gen=61] I190903 20:13:03.914175 59122 storage/replica_raft.go:291 [n1,s1,r90/2:/Table/70/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:03.916290 59252 storage/replica_command.go:1430 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r48:/Table/6{6/2-8} [(n3,s3):2, (n1,s1):3, next=4, gen=54] I190903 20:13:03.920184 59252 storage/replica_raft.go:291 [n1,s1,r48/3:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:03.929299 59269 storage/store.go:2593 [n3,replicaGC,s3,r90/1:/Table/70/1/{3/5-5}] removing replica r90/1 I190903 20:13:03.929732 59269 storage/replica_destroy.go:152 [n3,replicaGC,s3,r90/1:/Table/70/1/{3/5-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:03.930718 59316 storage/store.go:2593 [n3,replicaGC,s3,r48/2:/Table/6{6/2-8}] removing replica r48/2 I190903 20:13:03.931144 59316 storage/replica_destroy.go:152 [n3,replicaGC,s3,r48/2:/Table/6{6/2-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:04.019737 59557 storage/queue.go:1027 [n1,replicate,s1,r29/3:/Table/58{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:47] I190903 20:13:04.033235 59609 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/70/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r91:/Table/70/1/5{-/7} [(n3,s3):1, (n2,s2):2, next=3, gen=62] I190903 20:13:04.040450 59609 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:04.048785 59568 storage/store.go:2593 [n3,replicaGC,s3,r91/1:/Table/70/1/5{-/7}] removing replica r91/1 I190903 20:13:04.049187 59568 storage/replica_destroy.go:152 [n3,replicaGC,s3,r91/1:/Table/70/1/5{-/7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.054439 59408 storage/replica_command.go:1430 [n2,replicate,s2,r89/2:/Table/70/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r89:/Table/70/1/3/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=60] I190903 20:13:04.061628 59408 storage/replica_raft.go:291 [n2,s2,r89/2:/Table/70/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:04.110998 59857 storage/queue.go:1027 [n1,replicate,s1,r95/2:/Table/70/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 20:13:04.116947 59536 storage/store.go:2593 [n3,replicaGC,s3,r89/1:/Table/70/1/3/{4-5}] removing replica r89/1 I190903 20:13:04.117382 59536 storage/replica_destroy.go:152 [n3,replicaGC,s3,r89/1:/Table/70/1/3/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.144647 59962 storage/replica_command.go:1430 [n2,replicate,s2,r94/2:/Table/70/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r94:/Table/70/1/{5/9-6} [(n3,s3):1, (n2,s2):2, next=3, gen=65] I190903 20:13:04.150810 59962 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/70/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:04.159989 59980 storage/store.go:2593 [n3,replicaGC,s3,r94/1:/Table/70/1/{5/9-6}] removing replica r94/1 I190903 20:13:04.160382 59980 storage/replica_destroy.go:152 [n3,replicaGC,s3,r94/1:/Table/70/1/{5/9-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:04.221502 60119 storage/queue.go:1027 [n3,replicate,s3,r29/2:/Table/58{-/1}] no removable replicas from range that needs a removal: [2*:50, 3:0] E190903 20:13:04.225665 60157 storage/queue.go:1027 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] no removable replicas from range that needs a removal: [1*:26, 2:0] I190903 20:13:04.277414 60354 storage/replica_command.go:1430 [n1,replicate,s1,r95/2:/Table/70/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r95:/Table/70/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=66] I190903 20:13:04.282713 60354 storage/replica_raft.go:291 [n1,s1,r95/2:/Table/70/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:04.292003 60017 storage/store.go:2593 [n3,replicaGC,s3,r95/1:/Table/70/{1/6-2}] removing replica r95/1 I190903 20:13:04.292364 60017 storage/replica_destroy.go:152 [n3,replicaGC,s3,r95/1:/Table/70/{1/6-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.412815 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:04.414894 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:04.417397 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:04.421302 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:04.423241 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:04.427936 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:04.442525 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:04.444283 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:04.448776 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=70 (multi col list partitioning) version=2 mtime=2019-09-03 20:13:04.447375717 +0000 UTC I190903 20:13:04.481886 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 1}] I190903 20:13:04.483389 59084 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=2:1567541888.659875576,0, refcount=0 I190903 20:13:04.503164 61261 storage/replica_command.go:1430 [n3,replicate,s3,r96/1:/{Table/70/2-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r96:/{Table/70/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=66] I190903 20:13:04.513269 61261 storage/replica_raft.go:291 [n3,s3,r96/1:/{Table/70/2-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 20:13:04.525730 61303 storage/store.go:2593 [n1,replicaGC,s1,r96/2:/{Table/70/2-Max}] removing replica r96/2 I190903 20:13:04.526056 61303 storage/replica_destroy.go:152 [n1,replicaGC,s1,r96/2:/{Table/70/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.528146 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=70 (multi col list partitioning) version=3 mtime=2019-09-03 20:13:04.527930048 +0000 UTC I190903 20:13:04.533289 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 2}] I190903 20:13:04.539231 61353 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=3:1567541896.971908173,0, refcount=0 I190903 20:13:04.573038 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=70 (multi col list partitioning) version=4 mtime=2019-09-03 20:13:04.570834019 +0000 UTC I190903 20:13:04.579041 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 70, info: {TableName:data.public."multi col list partitioning" Statement:ALTER TABLE "multi col list partitioning" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p56 VALUES IN ((5, 6)), PARTITION p57 VALUES IN ((5, 7))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:04.581206 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 3}] I190903 20:13:04.582907 61630 sql/lease.go:923 new lease: 70("multi col list partitioning") ver=4:1567541896.972908173,0, refcount=0 E190903 20:13:04.586779 61582 storage/queue.go:1027 [n2,merge,s2,r94/2:/Table/70/1/{5/9-6}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):2 I190903 20:13:04.587698 61983 storage/replica_command.go:1430 [n2,s2,r94/2:/Table/70/1/{5/9-6}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r94:/Table/70/1/{5/9-6} [(n2,s2):2, next=3, gen=66] I190903 20:13:04.598722 61983 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:04.599876 61983 storage/store_snapshot.go:995 [n2,s2,r94/2:/Table/70/1/{5/9-6}] sending LEARNER snapshot a71d8521 at applied index 34 I190903 20:13:04.600047 61983 storage/store_snapshot.go:1038 [n2,s2,r94/2:/Table/70/1/{5/9-6}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:04.602960 58383 storage/replica_raftstorage.go:793 [n1,s1,r94/3:{-}] applying LEARNER snapshot [id=a71d8521 index=34] I190903 20:13:04.605215 58383 storage/replica_raftstorage.go:814 [n1,s1,r94/3:/Table/70/1/{5/9-6}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=a71d8521 index=34] I190903 20:13:04.605671 61983 storage/replica_command.go:1430 [n2,s2,r94/2:/Table/70/1/{5/9-6}] change replicas (add [(n1,s1):3] remove []): existing descriptor r94:/Table/70/1/{5/9-6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=67] I190903 20:13:04.616473 61983 storage/replica_raft.go:291 [n2,s2,r94/2:/Table/70/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 E190903 20:13:04.628257 62057 storage/queue.go:1027 [n1,replicate,s1,r94/3:/Table/70/1/{5/9-6}] no removable replicas from range that needs a removal: [2:0, 3*:38] I190903 20:13:04.640184 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:04.651178 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:04.676618 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p56 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:04.681758 62940 storage/replica_command.go:284 [n2,split,s2,r91/2:/Table/70/1/5{-/7}] initiating a split of this range at key /Table/70/1/5/6 [r56] (zone config) I190903 20:13:04.688485 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 70, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:04.689384 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:04.703909 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.709916 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.711150 63191 storage/replica_command.go:1430 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r56:/Table/70/1/5/{6-7} [(n2,s2):1, next=2, gen=64] I190903 20:13:04.715988 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.720982 62604 storage/replica_command.go:1430 [n1,s1,r94/3:/Table/70/1/{5/9-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r94:/Table/70/1/{5/9-6} [(n2,s2):2, (n1,s1):3, next=4, gen=68] I190903 20:13:04.724620 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.727748 62604 storage/replica_raft.go:291 [n1,s1,r94/3:/Table/70/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:04.735600 61819 storage/replica_command.go:609 [n1,merge,s1,r93/2:/Table/70/1/5/{8-9}] initiating a merge of r94:/Table/70/1/{5/9-6} [(n1,s1):3, next=4, gen=69] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:04.741914 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.744645 63191 storage/replica_raft.go:291 [n2,s2,r56/1:/Table/70/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:04.745700 63191 storage/store_snapshot.go:995 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] sending LEARNER snapshot 0cd88936 at applied index 15 I190903 20:13:04.745851 63191 storage/store_snapshot.go:1038 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:04.746514 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.746891 63379 storage/replica_raftstorage.go:793 [n3,s3,r56/2:{-}] applying LEARNER snapshot [id=0cd88936 index=15] I190903 20:13:04.748441 63379 storage/replica_raftstorage.go:814 [n3,s3,r56/2:/Table/70/1/5/{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=0cd88936 index=15] I190903 20:13:04.748948 63191 storage/replica_command.go:1430 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] change replicas (add [(n3,s3):2] remove []): existing descriptor r56:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=65] I190903 20:13:04.749089 63246 storage/store.go:2593 [n2,replicaGC,s2,r94/2:/Table/70/1/{5/9-6}] removing replica r94/2 I190903 20:13:04.749468 63246 storage/replica_destroy.go:152 [n2,replicaGC,s2,r94/2:/Table/70/1/{5/9-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.753452 211 storage/store.go:2593 [n1,s1,r93/2:/Table/70/1/5/{8-9}] removing replica r94/3 I190903 20:13:04.753619 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.755138 63520 storage/replica_command.go:1430 [n2,s2,r91/2:/Table/70/1/5{-/6}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, next=3, gen=64] I190903 20:13:04.755407 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.758838 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.761822 63486 storage/replica_command.go:1430 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r92:/Table/70/1/5/{7-8} [(n3,s3):1, next=2, gen=61] I190903 20:13:04.763054 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.763585 63191 storage/replica_raft.go:291 [n2,s2,r56/1:/Table/70/1/5/{6-7}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:04.780991 63191 storage/replica_command.go:1430 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r56:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):2, next=3, gen=66] I190903 20:13:04.782854 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.801218 63520 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/6}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:04.802667 63486 storage/replica_raft.go:291 [n3,s3,r92/1:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:04.803750 63486 storage/store_snapshot.go:995 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] sending LEARNER snapshot 80073aad at applied index 24 I190903 20:13:04.803943 63486 storage/store_snapshot.go:1038 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:04.804882 63191 storage/replica_raft.go:291 [n2,s2,r56/1:/Table/70/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:04.804919 63191 storage/replica_raft.go:304 [n2,s2,r56/1:/Table/70/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:04.805045 63561 storage/replica_raftstorage.go:793 [n1,s1,r92/2:{-}] applying LEARNER snapshot [id=80073aad index=24] I190903 20:13:04.805352 63520 storage/store_snapshot.go:995 [n2,s2,r91/2:/Table/70/1/5{-/6}] sending LEARNER snapshot 123773c2 at applied index 39 I190903 20:13:04.805523 63520 storage/store_snapshot.go:1038 [n2,s2,r91/2:/Table/70/1/5{-/6}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:04.805801 63191 storage/replica_command.go:979 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r56 failed: storage/replica_raft.go:305: [n2,s2,r56/1:/Table/70/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:04.805840 63191 storage/queue.go:1027 [n2,replicate,s2,r56/1:/Table/70/1/5/{6-7}] change replicas of r56 failed: storage/replica_raft.go:305: [n2,s2,r56/1:/Table/70/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:04.806381 63561 storage/replica_raftstorage.go:814 [n1,s1,r92/2:/Table/70/1/5/{7-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=80073aad index=24] I190903 20:13:04.807484 63486 storage/replica_command.go:1430 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] change replicas (add [(n1,s1):2] remove []): existing descriptor r92:/Table/70/1/5/{7-8} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=62] I190903 20:13:04.808854 63401 storage/replica_raftstorage.go:793 [n1,s1,r91/3:{-}] applying LEARNER snapshot [id=123773c2 index=39] I190903 20:13:04.809181 63732 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] change replicas (add [] remove [(n1,s1):3LEARNER]): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=65] I190903 20:13:04.810628 63401 storage/replica_raftstorage.go:814 [n1,s1,r91/3:/Table/70/1/5{-/6}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=123773c2 index=39] I190903 20:13:04.813808 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.822174 63486 storage/replica_raft.go:291 [n3,s3,r92/1:/Table/70/1/5/{7-8}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:04.822718 63486 storage/replica_command.go:1430 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r92:/Table/70/1/5/{7-8} [(n3,s3):1, (n1,s1):2, next=3, gen=63] I190903 20:13:04.836077 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.837549 63732 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/6}] proposing REMOVE_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2] next=4 I190903 20:13:04.839229 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.840825 63732 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] change replicas (add [(n1,s1):4LEARNER] remove []): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, next=4, gen=66] I190903 20:13:04.845460 63486 storage/replica_raft.go:291 [n3,s3,r92/1:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:04.845528 63486 storage/replica_raft.go:304 [n3,s3,r92/1:/Table/70/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:04.846400 63486 storage/replica_command.go:979 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r92 failed: storage/replica_raft.go:305: [n3,s3,r92/1:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:04.846439 63486 storage/queue.go:1027 [n3,replicate,s3,r92/1:/Table/70/1/5/{7-8}] change replicas of r92 failed: storage/replica_raft.go:305: [n3,s3,r92/1:/Table/70/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:04.846404 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.846949 63798 storage/store.go:2593 [n1,replicaGC,s1,r91/3:/Table/70/1/5{-/6}] removing replica r91/3 I190903 20:13:04.847387 63798 storage/replica_destroy.go:152 [n1,replicaGC,s1,r91/3:/Table/70/1/5{-/6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.847441 63727 storage/replica_command.go:1430 [n3,replicate,s3,r56/2:/Table/70/1/5/{6-7}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r56:/Table/70/1/5/{6-7} [(n2,s2):1, (n3,s3):2, next=3, gen=66] I190903 20:13:04.855326 63520 storage/replica_command.go:979 [n2,s2,r91/2:/Table/70/1/5{-/6}] could not promote [n1,s1] to voter, rolling back: change replicas of r91 failed: descriptor changed: [expected] r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=65] != [actual] r91:/Table/70/1/5{-/6} [(n2,s2):2, next=4, gen=66] E190903 20:13:04.855765 63470 storage/queue.go:1027 [n1,merge,s1,r90/2:/Table/70/1/{3/5-5}] while adding target n1,s1: change replicas of r91 failed: descriptor changed: [expected] r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=65] != [actual] r91:/Table/70/1/5{-/6} [(n2,s2):2, next=4, gen=66] I190903 20:13:04.856772 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:04.860638 63732 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/6}] proposing ADD_REPLICA[(n1,s1):4LEARNER]: after=[(n2,s2):2 (n1,s1):4LEARNER] next=5 I190903 20:13:04.861850 63727 storage/replica_raft.go:291 [n3,s3,r56/2:/Table/70/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 20:13:04.870788 63615 storage/replica_command.go:1430 [n3,replicate,s3,r29/2:/Table/58{-/1}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r29:/Table/58{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=18] I190903 20:13:04.880676 63732 storage/store_snapshot.go:995 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] sending LEARNER snapshot ce8221de at applied index 43 I190903 20:13:04.880910 63732 storage/store_snapshot.go:1038 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] streamed snapshot to (n1,s1):4LEARNER: kv pairs: 18, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 20:13:04.882672 63960 storage/replica_raftstorage.go:793 [n1,s1,r91/4:{-}] applying LEARNER snapshot [id=ce8221de index=43] I190903 20:13:04.886647 63960 storage/replica_raftstorage.go:814 [n1,s1,r91/4:/Table/70/1/5{-/6}] applied LEARNER snapshot [total=4ms ingestion=4@4ms id=ce8221de index=43] I190903 20:13:04.886692 63956 storage/store.go:2593 [n2,replicaGC,s2,r56/1:/Table/70/1/5/{6-7}] removing replica r56/1 I190903 20:13:04.887370 63732 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] change replicas (add [(n1,s1):4] remove []): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):4LEARNER, next=5, gen=67] I190903 20:13:04.894283 63956 storage/replica_destroy.go:152 [n2,replicaGC,s2,r56/1:/Table/70/1/5/{6-7}] removed 7 (0+7) keys in 7ms [clear=0ms commit=7ms] I190903 20:13:04.898619 63732 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/6}] proposing ADD_REPLICA[(n1,s1):4]: after=[(n2,s2):2 (n1,s1):4] next=5 I190903 20:13:04.899700 63732 storage/replica_command.go:1430 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):4, next=5, gen=68] I190903 20:13:04.908757 63615 storage/replica_raft.go:291 [n3,s3,r29/2:/Table/58{-/1}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n3,s3):2] next=4 I190903 20:13:04.919236 64097 storage/store.go:2593 [n1,replicaGC,s1,r29/3:/Table/58{-/1}] removing replica r29/3 I190903 20:13:04.920029 64097 storage/replica_destroy.go:152 [n1,replicaGC,s1,r29/3:/Table/58{-/1}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:04.920690 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:04.921690 63732 storage/replica_raft.go:291 [n2,s2,r91/2:/Table/70/1/5{-/6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):4] next=5 E190903 20:13:04.921777 63732 storage/replica_raft.go:304 [n2,s2,r91/2:/Table/70/1/5{-/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):4] next=5 to remove self (leaseholder) I190903 20:13:04.923235 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:04.924467 63732 storage/replica_command.go:979 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] could not promote [] to voter, rolling back: change replicas of r91 failed: storage/replica_raft.go:305: [n2,s2,r91/2:/Table/70/1/5{-/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):4] next=5 to remove self (leaseholder) E190903 20:13:04.924505 63732 storage/queue.go:1027 [n2,replicate,s2,r91/2:/Table/70/1/5{-/6}] change replicas of r91 failed: storage/replica_raft.go:305: [n2,s2,r91/2:/Table/70/1/5{-/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):4] next=5 to remove self (leaseholder) I190903 20:13:04.927299 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) E190903 20:13:04.927571 63715 storage/queue.go:1027 [n1,replicate,s1,r91/4:/Table/70/1/5{-/6}] no removable replicas from range that needs a removal: [2:0, 4*:49] I190903 20:13:04.984605 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:04.990358 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:04.995101 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:04.999318 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.005902 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.009699 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.016867 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.022963 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.033594 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:05.048430 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:05.051214 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:05.055115 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.057443 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.064369 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.081964 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:05.084001 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:05.088909 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.090946 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.094431 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.103756 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) I190903 20:13:05.105950 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:05.111642 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:05.117031 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.122139 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.132612 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.136917 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:05.149660 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:05.166262 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.169953 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.177290 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 I190903 20:13:05.188394 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) < (3, 4) I190903 20:13:05.198593 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 3 AND b = 4 I190903 20:13:05.205138 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (3, 4) AND (a, b) < (5, 6) I190903 20:13:05.226849 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 6 I190903 20:13:05.230133 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE a = 5 AND b = 7 E190903 20:13:05.233605 64592 storage/queue.go:1027 [n1,replicate,s1,r92/2:/Table/70/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:31] I190903 20:13:05.289169 52997 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning" WHERE (a, b) > (5, 7) === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT/multi_col_list_partitioning_-_DEFAULT_DEFAULT I190903 20:13:05.311068 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=70 (multi col list partitioning) version=5 mtime=2019-09-03 20:13:05.291864884 +0000 UTC I190903 20:13:05.318448 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 69, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning"]} I190903 20:13:05.322438 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning 70 4}] I190903 20:13:05.373784 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=70 (multi col list partitioning) version=6 mtime=2019-09-03 20:13:05.373620611 +0000 UTC I190903 20:13:05.444622 63954 storage/replica_command.go:1430 [n1,replicate,s1,r91/4:/Table/70/1/5{-/6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r91:/Table/70/1/5{-/6} [(n2,s2):2, (n1,s1):4, next=5, gen=68] I190903 20:13:05.456942 63954 storage/replica_raft.go:291 [n1,s1,r91/4:/Table/70/1/5{-/6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):4] next=5 I190903 20:13:05.461142 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 71, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:05.463226 66359 storage/replica_command.go:1430 [n1,replicate,s1,r92/2:/Table/70/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r92:/Table/70/1/5/{7-8} [(n3,s3):1, (n1,s1):2, next=3, gen=63] I190903 20:13:05.467997 66229 storage/store.go:2593 [n2,replicaGC,s2,r91/2:/Table/70/1/5{-/6}] removing replica r91/2 I190903 20:13:05.468400 66229 storage/replica_destroy.go:152 [n2,replicaGC,s2,r91/2:/Table/70/1/5{-/6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:05.478397 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 72, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root} I190903 20:13:05.487944 66359 storage/replica_raft.go:291 [n1,s1,r92/2:/Table/70/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:05.490100 66679 storage/replica_command.go:284 [n3,split,s3,r96/1:/{Table/70/2-Max}] initiating a split of this range at key /Table/72 [r97] (zone config) I190903 20:13:05.499294 65904 storage/store.go:2593 [n3,replicaGC,s3,r92/1:/Table/70/1/5/{7-8}] removing replica r92/1 I190903 20:13:05.499684 65904 storage/replica_destroy.go:152 [n3,replicaGC,s3,r92/1:/Table/70/1/5/{7-8}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:05.514316 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:05.525594 67198 storage/replica_command.go:284 [n3,split,s3,r97/1:/{Table/72-Max}] initiating a split of this range at key /Table/72/1 [r98] (zone config) I190903 20:13:05.537505 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:05.549427 67559 storage/replica_command.go:284 [n3,split,s3,r98/1:/{Table/72/1-Max}] initiating a split of this range at key /Table/72/1/3/4 [r99] (zone config) I190903 20:13:05.550648 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:05.561971 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:05.568027 67894 storage/replica_command.go:284 [n3,split,s3,r99/1:/{Table/72/1/3…-Max}] initiating a split of this range at key /Table/72/1/3/5 [r100] (zone config) I190903 20:13:05.569265 67751 storage/replica_command.go:1430 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r98:/Table/72/1{-/3/4} [(n3,s3):1, next=2, gen=70] I190903 20:13:05.593362 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:05.597826 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.607697 67751 storage/replica_raft.go:291 [n3,s3,r98/1:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:05.613903 67751 storage/store_snapshot.go:995 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] sending LEARNER snapshot 4ab106d6 at applied index 17 I190903 20:13:05.615405 67751 storage/store_snapshot.go:1038 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:05.617824 66781 storage/replica_raftstorage.go:793 [n1,s1,r98/2:{-}] applying LEARNER snapshot [id=4ab106d6 index=17] I190903 20:13:05.630117 68252 storage/replica_command.go:284 [n3,split,s3,r100/1:/{Table/72/1/3…-Max}] initiating a split of this range at key /Table/72/1/5 [r111] (zone config) I190903 20:13:05.632970 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:05.633948 66781 storage/replica_raftstorage.go:814 [n1,s1,r98/2:/Table/72/1{-/3/4}] applied LEARNER snapshot [total=16ms ingestion=4@15ms id=4ab106d6 index=17] I190903 20:13:05.634780 67751 storage/replica_command.go:1430 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r98:/Table/72/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=71] I190903 20:13:05.638916 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:05.667924 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.668295 67751 storage/replica_raft.go:291 [n3,s3,r98/1:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:05.674130 67751 storage/replica_command.go:1430 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r98:/Table/72/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=72] I190903 20:13:05.676536 68692 storage/replica_command.go:284 [n3,split,s3,r111/1:/{Table/72/1/5-Max}] initiating a split of this range at key /Table/72/1/5/7 [r112] (zone config) I190903 20:13:05.677585 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:05.686851 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:05.697361 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:05.697827 67751 storage/replica_raft.go:291 [n3,s3,r98/1:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:05.698440 67751 storage/replica_raft.go:304 [n3,s3,r98/1:/Table/72/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:05.703319 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.707125 67751 storage/replica_command.go:979 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r98 failed: storage/replica_raft.go:305: [n3,s3,r98/1:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:05.707183 67751 storage/queue.go:1027 [n3,replicate,s3,r98/1:/Table/72/1{-/3/4}] change replicas of r98 failed: storage/replica_raft.go:305: [n3,s3,r98/1:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:05.708464 68611 storage/replica_command.go:1430 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r99:/Table/72/1/3/{4-5} [(n3,s3):1, next=2, gen=71] I190903 20:13:05.719394 68854 storage/replica_command.go:284 [n3,split,s3,r112/1:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/5/8 [r113] (zone config) I190903 20:13:05.720467 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.741717 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.745224 68611 storage/replica_raft.go:291 [n3,s3,r99/1:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:05.748215 68611 storage/store_snapshot.go:995 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] sending LEARNER snapshot 2bd74d20 at applied index 18 I190903 20:13:05.748371 68611 storage/store_snapshot.go:1038 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:05.749447 69004 storage/replica_raftstorage.go:793 [n2,s2,r99/2:{-}] applying LEARNER snapshot [id=2bd74d20 index=18] I190903 20:13:05.750218 69029 storage/replica_command.go:284 [n3,split,s3,r113/1:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/5/9 [r114] (zone config) I190903 20:13:05.750814 69004 storage/replica_raftstorage.go:814 [n2,s2,r99/2:/Table/72/1/3/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=2bd74d20 index=18] I190903 20:13:05.751355 68611 storage/replica_command.go:1430 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r99:/Table/72/1/3/{4-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=72] I190903 20:13:05.751958 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:05.758752 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.758994 68715 storage/replica_command.go:609 [n1,merge,s1,r93/2:/Table/70/1/{5/8-6}] initiating a merge of r95:/Table/70/{1/6-2} [(n1,s1):2, next=3, gen=67] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:05.769038 185 storage/store.go:2593 [n1,s1,r93/2:/Table/70/1/{5/8-6}] removing replica r95/2 I190903 20:13:05.777885 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.783773 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.788997 69225 storage/replica_command.go:284 [n3,split,s3,r114/1:/{Table/72/1/5…-Max}] initiating a split of this range at key /Table/72/1/6 [r115] (zone config) I190903 20:13:05.791288 68611 storage/replica_raft.go:291 [n3,s3,r99/1:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:05.796043 68611 storage/replica_command.go:1430 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r99:/Table/72/1/3/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=73] I190903 20:13:05.801457 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.811759 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.815882 68611 storage/replica_raft.go:291 [n3,s3,r99/1:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:05.815929 68611 storage/replica_raft.go:304 [n3,s3,r99/1:/Table/72/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:05.816902 68611 storage/replica_command.go:979 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r99 failed: storage/replica_raft.go:305: [n3,s3,r99/1:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:05.816944 68611 storage/queue.go:1027 [n3,replicate,s3,r99/1:/Table/72/1/3/{4-5}] change replicas of r99 failed: storage/replica_raft.go:305: [n3,s3,r99/1:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:05.818952 69152 storage/replica_command.go:1430 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r100:/Table/72/1/{3/5-5} [(n3,s3):1, next=2, gen=72] E190903 20:13:05.819365 69366 storage/queue.go:1027 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:05.819818 69326 storage/replica_command.go:284 [n3,split,s3,r115/1:/{Table/72/1/6-Max}] initiating a split of this range at key /Table/72/2 [r116] (zone config) I190903 20:13:05.822253 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:05.831444 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:05.832236 69152 storage/replica_raft.go:291 [n3,s3,r100/1:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:05.839010 69152 storage/store_snapshot.go:995 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] sending LEARNER snapshot f49601ec at applied index 18 I190903 20:13:05.839166 69152 storage/store_snapshot.go:1038 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:05.840722 69090 storage/replica_raftstorage.go:793 [n1,s1,r100/2:{-}] applying LEARNER snapshot [id=f49601ec index=18] I190903 20:13:05.845951 69090 storage/replica_raftstorage.go:814 [n1,s1,r100/2:/Table/72/1/{3/5-5}] applied LEARNER snapshot [total=5ms ingestion=4@1ms id=f49601ec index=18] I190903 20:13:05.846873 69152 storage/replica_command.go:1430 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] change replicas (add [(n1,s1):2] remove []): existing descriptor r100:/Table/72/1/{3/5-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=73] I190903 20:13:05.872061 69152 storage/replica_raft.go:291 [n3,s3,r100/1:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:05.872616 69152 storage/replica_command.go:1430 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r100:/Table/72/1/{3/5-5} [(n3,s3):1, (n1,s1):2, next=3, gen=74] I190903 20:13:05.908062 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.908476 69152 storage/replica_raft.go:291 [n3,s3,r100/1:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:05.908597 69152 storage/replica_raft.go:304 [n3,s3,r100/1:/Table/72/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:05.911209 69152 storage/replica_command.go:979 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r100 failed: storage/replica_raft.go:305: [n3,s3,r100/1:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:05.911255 69152 storage/queue.go:1027 [n3,replicate,s3,r100/1:/Table/72/1/{3/5-5}] change replicas of r100 failed: storage/replica_raft.go:305: [n3,s3,r100/1:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:05.914553 69371 storage/replica_command.go:1430 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r111:/Table/72/1/5{-/7} [(n3,s3):1, next=2, gen=73] I190903 20:13:05.915798 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.921394 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:05.926594 69371 storage/replica_raft.go:291 [n3,s3,r111/1:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:05.926794 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.927824 69371 storage/store_snapshot.go:995 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] sending LEARNER snapshot 006891b2 at applied index 18 I190903 20:13:05.928007 69371 storage/store_snapshot.go:1038 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:05.929163 69712 storage/replica_raftstorage.go:793 [n2,s2,r111/2:{-}] applying LEARNER snapshot [id=006891b2 index=18] I190903 20:13:05.936143 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:05.939967 69712 storage/replica_raftstorage.go:814 [n2,s2,r111/2:/Table/72/1/5{-/7}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=006891b2 index=18] I190903 20:13:05.942682 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.944147 69371 storage/replica_command.go:1430 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r111:/Table/72/1/5{-/7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=74] I190903 20:13:05.951348 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.961642 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.965314 69371 storage/replica_raft.go:291 [n3,s3,r111/1:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:05.965899 69371 storage/replica_command.go:1430 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r111:/Table/72/1/5{-/7} [(n3,s3):1, (n2,s2):2, next=3, gen=75] I190903 20:13:05.974882 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:05.982206 69371 storage/replica_raft.go:291 [n3,s3,r111/1:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:05.982297 69371 storage/replica_raft.go:304 [n3,s3,r111/1:/Table/72/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:05.983145 69371 storage/replica_command.go:979 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r111 failed: storage/replica_raft.go:305: [n3,s3,r111/1:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:05.983192 69371 storage/queue.go:1027 [n3,replicate,s3,r111/1:/Table/72/1/5{-/7}] change replicas of r111 failed: storage/replica_raft.go:305: [n3,s3,r111/1:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:05.984366 69953 storage/replica_command.go:1430 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r113:/Table/72/1/5/{8-9} [(n3,s3):1, next=2, gen=75] I190903 20:13:05.996683 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:05.996995 69953 storage/replica_raft.go:291 [n3,s3,r113/1:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:05.999741 69953 storage/store_snapshot.go:995 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] sending LEARNER snapshot 88584706 at applied index 18 I190903 20:13:05.999901 69953 storage/store_snapshot.go:1038 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.007671 70116 storage/replica_raftstorage.go:793 [n1,s1,r113/2:{-}] applying LEARNER snapshot [id=88584706 index=18] I190903 20:13:06.015455 70116 storage/replica_raftstorage.go:814 [n1,s1,r113/2:/Table/72/1/5/{8-9}] applied LEARNER snapshot [total=8ms ingestion=4@7ms id=88584706 index=18] I190903 20:13:06.016564 69953 storage/replica_command.go:1430 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] change replicas (add [(n1,s1):2] remove []): existing descriptor r113:/Table/72/1/5/{8-9} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=76] I190903 20:13:06.037274 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:06.045614 69953 storage/replica_raft.go:291 [n3,s3,r113/1:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:06.046150 69953 storage/replica_command.go:1430 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r113:/Table/72/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=77] I190903 20:13:06.058513 69953 storage/replica_raft.go:291 [n3,s3,r113/1:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:06.058555 69953 storage/replica_raft.go:304 [n3,s3,r113/1:/Table/72/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:06.059565 69953 storage/replica_command.go:979 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r113 failed: storage/replica_raft.go:305: [n3,s3,r113/1:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:06.059598 69953 storage/queue.go:1027 [n3,replicate,s3,r113/1:/Table/72/1/5/{8-9}] change replicas of r113 failed: storage/replica_raft.go:305: [n3,s3,r113/1:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:06.061250 70326 storage/replica_command.go:1430 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r114:/Table/72/1/{5/9-6} [(n3,s3):1, next=2, gen=76] E190903 20:13:06.064683 70211 storage/queue.go:1027 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:06.069408 70326 storage/replica_raft.go:291 [n3,s3,r114/1:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:06.070641 70326 storage/store_snapshot.go:995 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] sending LEARNER snapshot a3ebb6b2 at applied index 18 I190903 20:13:06.070823 70326 storage/store_snapshot.go:1038 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.071912 70099 storage/replica_raftstorage.go:793 [n2,s2,r114/2:{-}] applying LEARNER snapshot [id=a3ebb6b2 index=18] I190903 20:13:06.073249 70099 storage/replica_raftstorage.go:814 [n2,s2,r114/2:/Table/72/1/{5/9-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a3ebb6b2 index=18] I190903 20:13:06.074421 70326 storage/replica_command.go:1430 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] change replicas (add [(n2,s2):2] remove []): existing descriptor r114:/Table/72/1/{5/9-6} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=77] I190903 20:13:06.083733 70326 storage/replica_raft.go:291 [n3,s3,r114/1:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:06.085911 70326 storage/replica_command.go:1430 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r114:/Table/72/1/{5/9-6} [(n3,s3):1, (n2,s2):2, next=3, gen=78] I190903 20:13:06.102540 70326 storage/replica_raft.go:291 [n3,s3,r114/1:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:06.102584 70326 storage/replica_raft.go:304 [n3,s3,r114/1:/Table/72/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:06.105695 70326 storage/replica_command.go:979 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r114 failed: storage/replica_raft.go:305: [n3,s3,r114/1:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:06.105739 70326 storage/queue.go:1027 [n3,replicate,s3,r114/1:/Table/72/1/{5/9-6}] change replicas of r114 failed: storage/replica_raft.go:305: [n3,s3,r114/1:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:06.108489 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 E190903 20:13:06.108655 70241 storage/queue.go:1027 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:06.109349 70500 storage/replica_command.go:1430 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r115:/Table/72/{1/6-2} [(n3,s3):1, next=2, gen=77] I190903 20:13:06.122169 70500 storage/replica_raft.go:291 [n3,s3,r115/1:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:06.123429 70500 storage/store_snapshot.go:995 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] sending LEARNER snapshot ca00c4fe at applied index 18 I190903 20:13:06.123583 70500 storage/store_snapshot.go:1038 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.124692 70486 storage/replica_raftstorage.go:793 [n1,s1,r115/2:{-}] applying LEARNER snapshot [id=ca00c4fe index=18] I190903 20:13:06.131837 70486 storage/replica_raftstorage.go:814 [n1,s1,r115/2:/Table/72/{1/6-2}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=ca00c4fe index=18] I190903 20:13:06.132563 70500 storage/replica_command.go:1430 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r115:/Table/72/{1/6-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=78] I190903 20:13:06.140612 70500 storage/replica_raft.go:291 [n3,s3,r115/1:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:06.141135 70500 storage/replica_command.go:1430 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r115:/Table/72/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=79] I190903 20:13:06.152410 70500 storage/replica_raft.go:291 [n3,s3,r115/1:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:06.153625 70500 storage/replica_raft.go:304 [n3,s3,r115/1:/Table/72/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:06.154864 70500 storage/replica_command.go:979 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r115 failed: storage/replica_raft.go:305: [n3,s3,r115/1:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:06.154938 70500 storage/queue.go:1027 [n3,replicate,s3,r115/1:/Table/72/{1/6-2}] change replicas of r115 failed: storage/replica_raft.go:305: [n3,s3,r115/1:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:06.164207 70402 storage/queue.go:1027 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:06.266119 71022 storage/queue.go:1027 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:06.294279 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.315034 71154 storage/replica_command.go:1430 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r98:/Table/72/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=72] I190903 20:13:06.324434 71154 storage/replica_raft.go:291 [n1,s1,r98/2:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:06.339489 71220 storage/store.go:2593 [n3,replicaGC,s3,r98/1:/Table/72/1{-/3/4}] removing replica r98/1 I190903 20:13:06.339858 71220 storage/replica_destroy.go:152 [n3,replicaGC,s3,r98/1:/Table/72/1{-/3/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:06.438299 71459 storage/queue.go:1027 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:06.566006 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.569279 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.570601 71904 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r114:/Table/72/1/{5/9-6} [(n3,s3):1, (n2,s2):2, next=3, gen=78] I190903 20:13:06.576491 71904 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:06.584597 71886 storage/store.go:2593 [n3,replicaGC,s3,r114/1:/Table/72/1/{5/9-6}] removing replica r114/1 I190903 20:13:06.584906 71886 storage/replica_destroy.go:152 [n3,replicaGC,s3,r114/1:/Table/72/1/{5/9-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:06.598059 71724 storage/replica_command.go:1430 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r100:/Table/72/1/{3/5-5} [(n3,s3):1, (n1,s1):2, next=3, gen=74] I190903 20:13:06.601538 71724 storage/replica_raft.go:291 [n1,s1,r100/2:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:06.609845 71930 storage/store.go:2593 [n3,replicaGC,s3,r100/1:/Table/72/1/{3/5-5}] removing replica r100/1 I190903 20:13:06.610127 71930 storage/replica_destroy.go:152 [n3,replicaGC,s3,r100/1:/Table/72/1/{3/5-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:06.612694 71932 storage/replica_command.go:1430 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r99:/Table/72/1/3/{4-5} [(n3,s3):1, (n2,s2):2, next=3, gen=73] I190903 20:13:06.619364 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:06.619553 71932 storage/replica_raft.go:291 [n2,s2,r99/2:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:06.622973 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:06.627089 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:06.627911 72089 storage/store.go:2593 [n3,replicaGC,s3,r99/1:/Table/72/1/3/{4-5}] removing replica r99/1 I190903 20:13:06.628987 72089 storage/replica_destroy.go:152 [n3,replicaGC,s3,r99/1:/Table/72/1/3/{4-5}] removed 9 (0+9) keys in 1ms [clear=1ms commit=0ms] I190903 20:13:06.629221 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:06.632255 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:06.633957 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:06.638150 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=2 mtime=2019-09-03 20:13:06.636836222 +0000 UTC I190903 20:13:06.645277 71999 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=2:1567541897.364878439,0, refcount=0 I190903 20:13:06.645774 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 1}] I190903 20:13:06.651303 72036 storage/replica_command.go:1430 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r113:/Table/72/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=77] I190903 20:13:06.655607 72036 storage/replica_raft.go:291 [n1,s1,r113/2:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:06.663843 72438 storage/store.go:2593 [n3,replicaGC,s3,r113/1:/Table/72/1/5/{8-9}] removing replica r113/1 I190903 20:13:06.664231 72438 storage/replica_destroy.go:152 [n3,replicaGC,s3,r113/1:/Table/72/1/5/{8-9}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:06.683291 72566 storage/replica_command.go:1430 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r111:/Table/72/1/5{-/7} [(n3,s3):1, (n2,s2):2, next=3, gen=75] I190903 20:13:06.689378 72566 storage/replica_raft.go:291 [n2,s2,r111/2:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:06.695068 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=3 mtime=2019-09-03 20:13:06.694885406 +0000 UTC I190903 20:13:06.697279 72441 storage/store.go:2593 [n3,replicaGC,s3,r111/1:/Table/72/1/5{-/7}] removing replica r111/1 I190903 20:13:06.697644 72441 storage/replica_destroy.go:152 [n3,replicaGC,s3,r111/1:/Table/72/1/5{-/7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:06.702345 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 2}] I190903 20:13:06.702749 72677 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=3:1567541897.365878439,0, refcount=0 I190903 20:13:06.725167 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=4 mtime=2019-09-03 20:13:06.724020555 +0000 UTC I190903 20:13:06.730584 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 72, info: {TableName:data.public."multi col list partitioning - DEFAULT DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT)), PARTITION pd VALUES IN ((DEFAULT, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:06.732551 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 3}] I190903 20:13:06.738946 72881 sql/lease.go:923 new lease: 72("multi col list partitioning - DEFAULT DEFAULT") ver=4:1567541897.366878439,0, refcount=0 I190903 20:13:06.759297 73326 storage/replica_command.go:609 [n1,merge,s1,r90/2:/Table/70/1/{3/5-5}] initiating a merge of r91:/Table/70/1/5{-/6} [(n1,s1):4, next=5, gen=69] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:06.765122 256 storage/store.go:2593 [n1,s1,r90/2:/Table/70/1/{3/5-5}] removing replica r91/4 I190903 20:13:06.771814 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:06.779657 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:06.788201 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:06.796303 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:06.805933 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 72, info: {Target:PARTITION pd OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:06.807019 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:06.815613 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:06.818703 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:06.820557 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.824161 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.826505 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.827652 74664 storage/replica_command.go:1430 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r115:/Table/72/{1/6-2} [(n3,s3):1, (n1,s1):2, next=3, gen=79] I190903 20:13:06.829608 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:06.831211 74664 storage/replica_raft.go:291 [n1,s1,r115/2:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:06.831629 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:06.832248 74664 storage/replica_command.go:1430 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r115:/Table/72/{1/6-2} [(n1,s1):2, next=3, gen=80] I190903 20:13:06.833628 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:06.838331 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:06.838945 74808 storage/store.go:2593 [n3,replicaGC,s3,r115/1:/Table/72/{1/6-2}] removing replica r115/1 I190903 20:13:06.839288 74808 storage/replica_destroy.go:152 [n3,replicaGC,s3,r115/1:/Table/72/{1/6-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:06.840350 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:06.841456 74664 storage/replica_raft.go:291 [n1,s1,r115/2:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:06.842577 74664 storage/store_snapshot.go:995 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] sending LEARNER snapshot d52f942e at applied index 30 I190903 20:13:06.842757 74664 storage/store_snapshot.go:1038 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.843504 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.843766 74670 storage/replica_raftstorage.go:793 [n2,s2,r115/3:{-}] applying LEARNER snapshot [id=d52f942e index=30] I190903 20:13:06.844994 74670 storage/replica_raftstorage.go:814 [n2,s2,r115/3:/Table/72/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d52f942e index=30] I190903 20:13:06.845529 74664 storage/replica_command.go:1430 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r115:/Table/72/{1/6-2} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=81] I190903 20:13:06.847403 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:06.850650 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:06.853865 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.854583 74664 storage/replica_raft.go:291 [n1,s1,r115/2:/Table/72/{1/6-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:06.855100 74664 storage/replica_command.go:1430 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r115:/Table/72/{1/6-2} [(n1,s1):2, (n2,s2):3, next=4, gen=82] I190903 20:13:06.857217 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:06.860651 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:06.863643 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:06.865184 74664 storage/replica_raft.go:291 [n1,s1,r115/2:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:06.865219 74664 storage/replica_raft.go:304 [n1,s1,r115/2:/Table/72/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:06.866277 74664 storage/replica_command.go:979 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r115 failed: storage/replica_raft.go:305: [n1,s1,r115/2:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:06.866327 74664 storage/queue.go:1027 [n1,replicate,s1,r115/2:/Table/72/{1/6-2}] change replicas of r115 failed: storage/replica_raft.go:305: [n1,s1,r115/2:/Table/72/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:06.866394 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:06.870089 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:06.873968 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.874919 74922 storage/replica_command.go:1430 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r112:/Table/72/1/5/{7-8} [(n3,s3):1, next=2, gen=74] I190903 20:13:06.878230 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.881871 74922 storage/replica_raft.go:291 [n3,s3,r112/1:/Table/72/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:06.882477 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:06.883013 74922 storage/store_snapshot.go:995 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] sending LEARNER snapshot ad4d3953 at applied index 20 I190903 20:13:06.883220 74922 storage/store_snapshot.go:1038 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.886146 74995 storage/replica_raftstorage.go:793 [n2,s2,r112/2:{-}] applying LEARNER snapshot [id=ad4d3953 index=20] I190903 20:13:06.887711 74995 storage/replica_raftstorage.go:814 [n2,s2,r112/2:/Table/72/1/5/{7-8}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=ad4d3953 index=20] I190903 20:13:06.888216 74922 storage/replica_command.go:1430 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] change replicas (add [(n2,s2):2] remove []): existing descriptor r112:/Table/72/1/5/{7-8} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=75] I190903 20:13:06.889637 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:06.901068 74922 storage/replica_raft.go:291 [n3,s3,r112/1:/Table/72/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:06.901662 74922 storage/replica_command.go:1430 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r112:/Table/72/1/5/{7-8} [(n3,s3):1, (n2,s2):2, next=3, gen=76] I190903 20:13:06.906110 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:06.911823 74922 storage/replica_raft.go:291 [n3,s3,r112/1:/Table/72/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:06.911866 74922 storage/replica_raft.go:304 [n3,s3,r112/1:/Table/72/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:06.912705 74922 storage/replica_command.go:979 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r112 failed: storage/replica_raft.go:305: [n3,s3,r112/1:/Table/72/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:06.912742 74922 storage/queue.go:1027 [n3,replicate,s3,r112/1:/Table/72/1/5/{7-8}] change replicas of r112 failed: storage/replica_raft.go:305: [n3,s3,r112/1:/Table/72/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:06.919592 75144 storage/replica_command.go:1430 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r100:/Table/72/1/{3/5-5} [(n1,s1):2, next=3, gen=75] I190903 20:13:06.923103 75144 storage/replica_raft.go:291 [n1,s1,r100/2:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:06.925067 75144 storage/store_snapshot.go:995 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] sending LEARNER snapshot fb4f7ccd at applied index 32 I190903 20:13:06.925316 75144 storage/store_snapshot.go:1038 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:06.926460 70433 storage/replica_raftstorage.go:793 [n2,s2,r100/3:{-}] applying LEARNER snapshot [id=fb4f7ccd index=32] I190903 20:13:06.928293 70433 storage/replica_raftstorage.go:814 [n2,s2,r100/3:/Table/72/1/{3/5-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=fb4f7ccd index=32] I190903 20:13:06.928872 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:06.928969 75144 storage/replica_command.go:1430 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r100:/Table/72/1/{3/5-5} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=76] I190903 20:13:06.932633 75144 storage/replica_raft.go:291 [n1,s1,r100/2:/Table/72/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:06.933092 75144 storage/replica_command.go:1430 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r100:/Table/72/1/{3/5-5} [(n1,s1):2, (n2,s2):3, next=4, gen=77] I190903 20:13:06.943919 75144 storage/replica_raft.go:291 [n1,s1,r100/2:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:06.943951 75144 storage/replica_raft.go:304 [n1,s1,r100/2:/Table/72/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:06.944814 75144 storage/replica_command.go:979 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r100 failed: storage/replica_raft.go:305: [n1,s1,r100/2:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:06.944856 75144 storage/queue.go:1027 [n1,replicate,s1,r100/2:/Table/72/1/{3/5-5}] change replicas of r100 failed: storage/replica_raft.go:305: [n1,s1,r100/2:/Table/72/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:06.966621 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:07.006251 75528 storage/replica_command.go:1430 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r111:/Table/72/1/5{-/7} [(n2,s2):2, next=3, gen=76] I190903 20:13:07.015425 75528 storage/replica_raft.go:291 [n2,s2,r111/2:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:07.016695 75528 storage/store_snapshot.go:995 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] sending LEARNER snapshot 6a9b376e at applied index 35 I190903 20:13:07.016925 75528 storage/store_snapshot.go:1038 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:07.018872 75547 storage/replica_raftstorage.go:793 [n1,s1,r111/3:{-}] applying LEARNER snapshot [id=6a9b376e index=35] I190903 20:13:07.030702 75547 storage/replica_raftstorage.go:814 [n1,s1,r111/3:/Table/72/1/5{-/7}] applied LEARNER snapshot [total=12ms ingestion=4@11ms id=6a9b376e index=35] I190903 20:13:07.031511 75528 storage/replica_command.go:1430 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r111:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=77] I190903 20:13:07.037103 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:07.040012 75528 storage/replica_raft.go:291 [n2,s2,r111/2:/Table/72/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:07.040540 75528 storage/replica_command.go:1430 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r111:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3, next=4, gen=78] I190903 20:13:07.054734 75528 storage/replica_raft.go:291 [n2,s2,r111/2:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:07.054782 75528 storage/replica_raft.go:304 [n2,s2,r111/2:/Table/72/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.055581 75528 storage/replica_command.go:979 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r111 failed: storage/replica_raft.go:305: [n2,s2,r111/2:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:07.055620 75528 storage/queue.go:1027 [n2,replicate,s2,r111/2:/Table/72/1/5{-/7}] change replicas of r111 failed: storage/replica_raft.go:305: [n2,s2,r111/2:/Table/72/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.056004 75697 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r114:/Table/72/1/{5/9-6} [(n2,s2):2, next=3, gen=79] I190903 20:13:07.058774 75579 storage/replica_command.go:1430 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r98:/Table/72/1{-/3/4} [(n1,s1):2, next=3, gen=73] I190903 20:13:07.062337 75579 storage/replica_raft.go:291 [n1,s1,r98/2:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:07.063596 75579 storage/store_snapshot.go:995 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] sending LEARNER snapshot 29521329 at applied index 32 I190903 20:13:07.066280 75579 storage/store_snapshot.go:1038 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:07.067563 75762 storage/replica_raftstorage.go:793 [n2,s2,r98/3:{-}] applying LEARNER snapshot [id=29521329 index=32] I190903 20:13:07.067658 75697 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:07.071130 75762 storage/replica_raftstorage.go:814 [n2,s2,r98/3:/Table/72/1{-/3/4}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=29521329 index=32] I190903 20:13:07.071576 75697 storage/store_snapshot.go:995 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] sending LEARNER snapshot acecee58 at applied index 35 I190903 20:13:07.071758 75697 storage/store_snapshot.go:1038 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:07.071976 75579 storage/replica_command.go:1430 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] change replicas (add [(n2,s2):3] remove []): existing descriptor r98:/Table/72/1{-/3/4} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=74] I190903 20:13:07.075403 75786 storage/replica_raftstorage.go:793 [n1,s1,r114/3:{-}] applying LEARNER snapshot [id=acecee58 index=35] I190903 20:13:07.076175 75579 storage/replica_raft.go:291 [n1,s1,r98/2:/Table/72/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:07.078213 75579 storage/replica_command.go:1430 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r98:/Table/72/1{-/3/4} [(n1,s1):2, (n2,s2):3, next=4, gen=75] I190903 20:13:07.079604 75786 storage/replica_raftstorage.go:814 [n1,s1,r114/3:/Table/72/1/{5/9-6}] applied LEARNER snapshot [total=4ms ingestion=4@2ms id=acecee58 index=35] I190903 20:13:07.080219 75697 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] change replicas (add [(n1,s1):3] remove []): existing descriptor r114:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=80] I190903 20:13:07.089431 75697 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:07.090050 75697 storage/replica_command.go:1430 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r114:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3, next=4, gen=81] I190903 20:13:07.093367 75579 storage/replica_raft.go:291 [n1,s1,r98/2:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:07.093407 75579 storage/replica_raft.go:304 [n1,s1,r98/2:/Table/72/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:07.098857 75579 storage/replica_command.go:979 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r98 failed: storage/replica_raft.go:305: [n1,s1,r98/2:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:07.098935 75579 storage/queue.go:1027 [n1,replicate,s1,r98/2:/Table/72/1{-/3/4}] change replicas of r98 failed: storage/replica_raft.go:305: [n1,s1,r98/2:/Table/72/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:07.103025 75697 storage/replica_raft.go:291 [n2,s2,r114/2:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:07.103074 75697 storage/replica_raft.go:304 [n2,s2,r114/2:/Table/72/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.103934 75697 storage/replica_command.go:979 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r114 failed: storage/replica_raft.go:305: [n2,s2,r114/2:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:07.103967 75697 storage/queue.go:1027 [n2,replicate,s2,r114/2:/Table/72/1/{5/9-6}] change replicas of r114 failed: storage/replica_raft.go:305: [n2,s2,r114/2:/Table/72/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.104403 75904 storage/replica_command.go:1430 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r99:/Table/72/1/3/{4-5} [(n2,s2):2, next=3, gen=74] I190903 20:13:07.116492 75904 storage/replica_raft.go:291 [n2,s2,r99/2:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:07.117457 75904 storage/store_snapshot.go:995 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] sending LEARNER snapshot 34c4efca at applied index 34 I190903 20:13:07.117674 75904 storage/store_snapshot.go:1038 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:07.118868 75966 storage/replica_raftstorage.go:793 [n1,s1,r99/3:{-}] applying LEARNER snapshot [id=34c4efca index=34] I190903 20:13:07.120167 75966 storage/replica_raftstorage.go:814 [n1,s1,r99/3:/Table/72/1/3/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=34c4efca index=34] I190903 20:13:07.120636 75904 storage/replica_command.go:1430 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r99:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=75] I190903 20:13:07.132823 75904 storage/replica_raft.go:291 [n2,s2,r99/2:/Table/72/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:07.135356 75904 storage/replica_command.go:1430 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r99:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3, next=4, gen=76] I190903 20:13:07.154516 75904 storage/replica_raft.go:291 [n2,s2,r99/2:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:07.154601 75904 storage/replica_raft.go:304 [n2,s2,r99/2:/Table/72/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.160916 75904 storage/replica_command.go:979 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r99 failed: storage/replica_raft.go:305: [n2,s2,r99/2:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:07.161063 75904 storage/queue.go:1027 [n2,replicate,s2,r99/2:/Table/72/1/3/{4-5}] change replicas of r99 failed: storage/replica_raft.go:305: [n2,s2,r99/2:/Table/72/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:07.161975 75928 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [(n3,s3):5LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, next=5, gen=9] I190903 20:13:07.174363 75928 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):5LEARNER]: after=[(n2,s2):4 (n3,s3):5LEARNER] next=6 I190903 20:13:07.175247 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:07.175342 75928 storage/store_snapshot.go:995 [n2,replicate,s2,r4/4:/System{/tsd-tse}] sending LEARNER snapshot 04734e12 at applied index 93 I190903 20:13:07.195826 75928 storage/store_snapshot.go:1038 [n2,replicate,s2,r4/4:/System{/tsd-tse}] streamed snapshot to (n3,s3):5LEARNER: kv pairs: 1958, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 20:13:07.199837 76193 storage/replica_raftstorage.go:793 [n3,s3,r4/5:{-}] applying LEARNER snapshot [id=04734e12 index=93] I190903 20:13:07.202241 76193 storage/replica_raftstorage.go:814 [n3,s3,r4/5:/System{/tsd-tse}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=04734e12 index=93] I190903 20:13:07.203079 75928 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [(n3,s3):5] remove []): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5LEARNER, next=6, gen=10] I190903 20:13:07.220112 75928 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing ADD_REPLICA[(n3,s3):5]: after=[(n2,s2):4 (n3,s3):5] next=6 I190903 20:13:07.221982 75928 storage/replica_command.go:1430 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5, next=6, gen=11] I190903 20:13:07.232856 75928 storage/replica_raft.go:291 [n2,s2,r4/4:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 E190903 20:13:07.232899 75928 storage/replica_raft.go:304 [n2,s2,r4/4:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) I190903 20:13:07.233855 75928 storage/replica_command.go:979 [n2,replicate,s2,r4/4:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/4:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) E190903 20:13:07.233887 75928 storage/queue.go:1027 [n2,replicate,s2,r4/4:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n2,s2,r4/4:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 to remove self (leaseholder) E190903 20:13:07.237719 76426 storage/queue.go:1027 [n1,replicate,s1,r114/3:/Table/72/1/{5/9-6}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 20:13:07.238170 76367 storage/replica_command.go:1430 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r113:/Table/72/1/5/{8-9} [(n1,s1):2, next=3, gen=78] I190903 20:13:07.242066 76367 storage/replica_raft.go:291 [n1,s1,r113/2:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:07.243345 76367 storage/store_snapshot.go:995 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] sending LEARNER snapshot e1bab89d at applied index 33 I190903 20:13:07.243545 76367 storage/store_snapshot.go:1038 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:07.244820 76470 storage/replica_raftstorage.go:793 [n3,s3,r113/3:{-}] applying LEARNER snapshot [id=e1bab89d index=33] I190903 20:13:07.256411 76470 storage/replica_raftstorage.go:814 [n3,s3,r113/3:/Table/72/1/5/{8-9}] applied LEARNER snapshot [total=12ms ingestion=4@11ms id=e1bab89d index=33] I190903 20:13:07.257059 76367 storage/replica_command.go:1430 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] change replicas (add [(n3,s3):3] remove []): existing descriptor r113:/Table/72/1/5/{8-9} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=79] I190903 20:13:07.260664 76367 storage/replica_raft.go:291 [n1,s1,r113/2:/Table/72/1/5/{8-9}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 20:13:07.261278 76367 storage/replica_command.go:1430 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r113:/Table/72/1/5/{8-9} [(n1,s1):2, (n3,s3):3, next=4, gen=80] I190903 20:13:07.270598 76367 storage/replica_raft.go:291 [n1,s1,r113/2:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 20:13:07.270644 76367 storage/replica_raft.go:304 [n1,s1,r113/2:/Table/72/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:07.271512 76367 storage/replica_command.go:979 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r113 failed: storage/replica_raft.go:305: [n1,s1,r113/2:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:07.271559 76367 storage/queue.go:1027 [n1,replicate,s1,r113/2:/Table/72/1/5/{8-9}] change replicas of r113 failed: storage/replica_raft.go:305: [n1,s1,r113/2:/Table/72/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:07.295388 76409 storage/replica_command.go:1430 [n2,replicate,s2,r100/3:/Table/72/1/{3/5-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r100:/Table/72/1/{3/5-5} [(n1,s1):2, (n2,s2):3, next=4, gen=77] I190903 20:13:07.308885 76409 storage/replica_raft.go:291 [n2,s2,r100/3:/Table/72/1/{3/5-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:07.311737 76448 storage/replica_command.go:1430 [n2,replicate,s2,r115/3:/Table/72/{1/6-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r115:/Table/72/{1/6-2} [(n1,s1):2, (n2,s2):3, next=4, gen=82] E190903 20:13:07.312235 76476 storage/queue.go:1027 [n1,replicate,s1,r99/3:/Table/72/1/3/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 20:13:07.325390 76448 storage/replica_raft.go:291 [n2,s2,r115/3:/Table/72/{1/6-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:07.335210 76316 storage/replica_command.go:1430 [n2,replicate,s2,r112/2:/Table/72/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r112:/Table/72/1/5/{7-8} [(n3,s3):1, (n2,s2):2, next=3, gen=76] I190903 20:13:07.344146 76316 storage/replica_raft.go:291 [n2,s2,r112/2:/Table/72/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:07.384121 76445 storage/store.go:2593 [n1,replicaGC,s1,r100/2:/Table/72/1/{3/5-5}] removing replica r100/2 I190903 20:13:07.385233 76445 storage/replica_destroy.go:152 [n1,replicaGC,s1,r100/2:/Table/72/1/{3/5-5}] removed 8 (0+8) keys in 1ms [clear=0ms commit=1ms] I190903 20:13:07.385515 76721 storage/store.go:2593 [n1,replicaGC,s1,r115/2:/Table/72/{1/6-2}] removing replica r115/2 I190903 20:13:07.386037 76721 storage/replica_destroy.go:152 [n1,replicaGC,s1,r115/2:/Table/72/{1/6-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:07.404960 76821 storage/store.go:2593 [n3,replicaGC,s3,r112/1:/Table/72/1/5/{7-8}] removing replica r112/1 I190903 20:13:07.405278 76821 storage/replica_destroy.go:152 [n3,replicaGC,s3,r112/1:/Table/72/1/5/{7-8}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:07.411957 77004 storage/queue.go:1027 [n3,replicate,s3,r4/5:/System{/tsd-tse}] no removable replicas from range that needs a removal: [4:0, 5*:100] I190903 20:13:07.447946 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:07.452260 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:07.471268 293 storage/compactor/compactor.go:370 [n1,s1,compactor] processing compaction #1-38/38 (/System/tsd-/Max) for 91 KiB (reasons: size=false used=true avail=false) I190903 20:13:07.501378 293 storage/compactor/compactor.go:386 [n1,s1,compactor] processed compaction #1-38/38 (/System/tsd-/Max) for 91 KiB in 0.0s I190903 20:13:07.525637 77302 storage/replica_command.go:1430 [n1,replicate,s1,r114/3:/Table/72/1/{5/9-6}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r114:/Table/72/1/{5/9-6} [(n2,s2):2, (n1,s1):3, next=4, gen=81] I190903 20:13:07.537680 77302 storage/replica_raft.go:291 [n1,s1,r114/3:/Table/72/1/{5/9-6}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:07.551429 77351 storage/store.go:2593 [n2,replicaGC,s2,r114/2:/Table/72/1/{5/9-6}] removing replica r114/2 I190903 20:13:07.551998 77351 storage/replica_destroy.go:152 [n2,replicaGC,s2,r114/2:/Table/72/1/{5/9-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:07.585804 77320 storage/replica_command.go:1430 [n1,replicate,s1,r99/3:/Table/72/1/3/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r99:/Table/72/1/3/{4-5} [(n2,s2):2, (n1,s1):3, next=4, gen=76] I190903 20:13:07.590011 77320 storage/replica_raft.go:291 [n1,s1,r99/3:/Table/72/1/3/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:07.595600 77502 storage/replica_command.go:1430 [n1,replicate,s1,r111/3:/Table/72/1/5{-/7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r111:/Table/72/1/5{-/7} [(n2,s2):2, (n1,s1):3, next=4, gen=78] I190903 20:13:07.598895 77502 storage/replica_raft.go:291 [n1,s1,r111/3:/Table/72/1/5{-/7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:07.602431 77557 storage/queue.go:1027 [n3,replicate,s3,r113/3:/Table/72/1/5/{8-9}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 20:13:07.606588 77480 storage/store.go:2593 [n2,replicaGC,s2,r99/2:/Table/72/1/3/{4-5}] removing replica r99/2 I190903 20:13:07.607023 77480 storage/replica_destroy.go:152 [n2,replicaGC,s2,r99/2:/Table/72/1/3/{4-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:07.607899 77530 storage/store.go:2593 [n2,replicaGC,s2,r111/2:/Table/72/1/5{-/7}] removing replica r111/2 I190903 20:13:07.608332 77530 storage/replica_destroy.go:152 [n2,replicaGC,s2,r111/2:/Table/72/1/5{-/7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:07.723848 77536 storage/queue.go:1027 [n2,replicate,s2,r98/3:/Table/72/1{-/3/4}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 20:13:07.822364 78112 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [] remove [(n2,s2):4]): existing descriptor r4:/System{/tsd-tse} [(n2,s2):4, (n3,s3):5, next=6, gen=11] I190903 20:13:07.828696 78112 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n2,s2):4]: after=[(n3,s3):5] next=6 I190903 20:13:07.861894 78293 storage/replica_command.go:1430 [n2,replicate,s2,r98/3:/Table/72/1{-/3/4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r98:/Table/72/1{-/3/4} [(n1,s1):2, (n2,s2):3, next=4, gen=75] I190903 20:13:07.868528 78293 storage/replica_raft.go:291 [n2,s2,r98/3:/Table/72/1{-/3/4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:07.885302 78213 storage/store.go:2593 [n2,replicaGC,s2,r4/4:/System{/tsd-tse}] removing replica r4/4 I190903 20:13:07.887299 78213 storage/replica_destroy.go:152 [n2,replicaGC,s2,r4/4:/System{/tsd-tse}] removed 1950 (1941+9) keys in 1ms [clear=1ms commit=0ms] I190903 20:13:07.905504 78437 storage/replica_command.go:1430 [n3,replicate,s3,r113/3:/Table/72/1/5/{8-9}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r113:/Table/72/1/5/{8-9} [(n1,s1):2, (n3,s3):3, next=4, gen=80] I190903 20:13:07.913398 78437 storage/replica_raft.go:291 [n3,s3,r113/3:/Table/72/1/5/{8-9}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 20:13:07.935352 78326 storage/store.go:2593 [n1,replicaGC,s1,r98/2:/Table/72/1{-/3/4}] removing replica r98/2 I190903 20:13:07.935760 78326 storage/replica_destroy.go:152 [n1,replicaGC,s1,r98/2:/Table/72/1{-/3/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:07.936081 78140 storage/store.go:2593 [n1,replicaGC,s1,r113/2:/Table/72/1/5/{8-9}] removing replica r113/2 I190903 20:13:07.936368 78140 storage/replica_destroy.go:152 [n1,replicaGC,s1,r113/2:/Table/72/1/5/{8-9}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:07.996970 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:08.000210 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:08.004327 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.007418 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:08.010643 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:08.013251 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.016764 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:08.020511 65247 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 === RUN TestRepartitioning/multi_col_list_partitioning_-_DEFAULT_DEFAULT/multi_col_list_partitioning_-_DEFAULT I190903 20:13:08.033561 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=5 mtime=2019-09-03 20:13:08.024397567 +0000 UTC I190903 20:13:08.041081 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 71, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT DEFAULT"]} I190903 20:13:08.043144 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT DEFAULT 72 4}] I190903 20:13:08.091827 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=72 (multi col list partitioning - DEFAULT DEFAULT) version=6 mtime=2019-09-03 20:13:08.09166272 +0000 UTC I190903 20:13:08.108311 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 73, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:08.113581 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 74, info: {TableName:data.public."multi col list partitioning - DEFAULT DEFAULT" Statement:CREATE TABLE "multi col list partitioning - DEFAULT DEFAULT" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT)), PARTITION pd VALUES IN ((DEFAULT, DEFAULT))) User:root} I190903 20:13:08.114938 80065 storage/replica_command.go:284 [n3,split,s3,r116/1:/{Table/72/2-Max}] initiating a split of this range at key /Table/74 [r117] (zone config) I190903 20:13:08.121751 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:08.123626 2316 storage/queue.go:518 [n3,s3,r116/1:/Table/7{2/2-4}] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:08.142721 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:08.145878 80600 storage/replica_command.go:284 [n3,split,s3,r117/1:/{Table/74-Max}] initiating a split of this range at key /Table/74/1/3/4 [r118] (zone config) I190903 20:13:08.155015 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:08.156744 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:08.161886 80304 storage/replica_command.go:284 [n3,split,s3,r118/1:/{Table/74/1/3…-Max}] initiating a split of this range at key /Table/74/1/3/5 [r119] (zone config) I190903 20:13:08.164113 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:08.171696 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION pd OF INDEX data.public."multi col list partitioning - DEFAULT DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:08.171816 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:08.174375 80338 storage/replica_command.go:284 [n3,split,s3,r117/1:/Table/74{-/1/3/4}] initiating a split of this range at key /Table/74/1 [r120] (zone config) I190903 20:13:08.179123 81532 storage/replica_command.go:284 [n3,split,s3,r119/1:/{Table/74/1/3…-Max}] initiating a split of this range at key /Table/74/1/5 [r121] (zone config) I190903 20:13:08.179546 81189 storage/replica_command.go:1430 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r118:/Table/74/1/3/{4-5} [(n3,s3):1, next=2, gen=80] I190903 20:13:08.184645 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:08.188304 81189 storage/replica_raft.go:291 [n3,s3,r118/1:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:08.188909 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:08.189691 81189 storage/store_snapshot.go:995 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] sending LEARNER snapshot 876cab72 at applied index 17 I190903 20:13:08.189858 81189 storage/store_snapshot.go:1038 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.190870 81673 storage/replica_raftstorage.go:793 [n1,s1,r118/2:{-}] applying LEARNER snapshot [id=876cab72 index=17] I190903 20:13:08.191269 81717 storage/replica_command.go:284 [n3,split,s3,r121/1:/{Table/74/1/5-Max}] initiating a split of this range at key /Table/74/1/5/7 [r122] (zone config) I190903 20:13:08.193607 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.197635 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:08.198796 81673 storage/replica_raftstorage.go:814 [n1,s1,r118/2:/Table/74/1/3/{4-5}] applied LEARNER snapshot [total=8ms ingestion=4@8ms id=876cab72 index=17] I190903 20:13:08.199243 81189 storage/replica_command.go:1430 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] change replicas (add [(n1,s1):2] remove []): existing descriptor r118:/Table/74/1/3/{4-5} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=81] I190903 20:13:08.201211 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.201289 81728 storage/replica_command.go:284 [n3,split,s3,r122/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/5/8 [r123] (zone config) I190903 20:13:08.204276 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:08.206150 81189 storage/replica_raft.go:291 [n3,s3,r118/1:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:08.206725 81189 storage/replica_command.go:1430 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r118:/Table/74/1/3/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=82] I190903 20:13:08.207501 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:08.210731 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.214266 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.216850 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.219871 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:08.220012 81189 storage/replica_raft.go:291 [n3,s3,r118/1:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:08.220054 81189 storage/replica_raft.go:304 [n3,s3,r118/1:/Table/74/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.220846 81915 storage/replica_command.go:284 [n3,split,s3,r123/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/5/9 [r124] (zone config) I190903 20:13:08.220978 81189 storage/replica_command.go:979 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r118 failed: storage/replica_raft.go:305: [n3,s3,r118/1:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:08.221020 81189 storage/queue.go:1027 [n3,replicate,s3,r118/1:/Table/74/1/3/{4-5}] change replicas of r118 failed: storage/replica_raft.go:305: [n3,s3,r118/1:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.221549 81901 storage/replica_command.go:1430 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r120:/Table/74/1{-/3/4} [(n3,s3):1, next=2, gen=80] I190903 20:13:08.224224 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:08.227238 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.228957 81901 storage/replica_raft.go:291 [n3,s3,r120/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:08.229930 81901 storage/store_snapshot.go:995 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] sending LEARNER snapshot b385164a at applied index 14 I190903 20:13:08.230067 81901 storage/store_snapshot.go:1038 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.230176 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:08.231528 81808 storage/replica_raftstorage.go:793 [n2,s2,r120/2:{-}] applying LEARNER snapshot [id=b385164a index=14] I190903 20:13:08.231584 81997 storage/replica_command.go:284 [n3,split,s3,r124/1:/{Table/74/1/5…-Max}] initiating a split of this range at key /Table/74/1/6 [r125] (zone config) I190903 20:13:08.232465 81808 storage/replica_raftstorage.go:814 [n2,s2,r120/2:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=b385164a index=14] I190903 20:13:08.232947 81901 storage/replica_command.go:1430 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r120:/Table/74/1{-/3/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=81] I190903 20:13:08.234412 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:08.238634 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:08.240276 81901 storage/replica_raft.go:291 [n3,s3,r120/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:08.240783 81901 storage/replica_command.go:1430 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r120:/Table/74/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=82] I190903 20:13:08.242278 82067 storage/replica_command.go:284 [n3,split,s3,r125/1:/{Table/74/1/6-Max}] initiating a split of this range at key /Table/74/2 [r126] (zone config) I190903 20:13:08.242405 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:08.246635 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:08.250740 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:08.260753 81901 storage/replica_raft.go:291 [n3,s3,r120/1:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:08.260800 81901 storage/replica_raft.go:304 [n3,s3,r120/1:/Table/74/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.261874 81901 storage/replica_command.go:979 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r120 failed: storage/replica_raft.go:305: [n3,s3,r120/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:08.261905 81901 storage/queue.go:1027 [n3,replicate,s3,r120/1:/Table/74/1{-/3/4}] change replicas of r120 failed: storage/replica_raft.go:305: [n3,s3,r120/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.262250 82029 storage/replica_command.go:1430 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r119:/Table/74/1/{3/5-5} [(n3,s3):1, next=2, gen=81] I190903 20:13:08.265608 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.269442 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.270400 82029 storage/replica_raft.go:291 [n3,s3,r119/1:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:08.271427 82029 storage/store_snapshot.go:995 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] sending LEARNER snapshot d72e820b at applied index 18 I190903 20:13:08.271580 82029 storage/store_snapshot.go:1038 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.273372 82273 storage/replica_raftstorage.go:793 [n2,s2,r119/2:{-}] applying LEARNER snapshot [id=d72e820b index=18] I190903 20:13:08.274179 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.274590 82273 storage/replica_raftstorage.go:814 [n2,s2,r119/2:/Table/74/1/{3/5-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d72e820b index=18] I190903 20:13:08.275017 82029 storage/replica_command.go:1430 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r119:/Table/74/1/{3/5-5} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=82] I190903 20:13:08.288702 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:08.291795 82029 storage/replica_raft.go:291 [n3,s3,r119/1:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:08.292375 82029 storage/replica_command.go:1430 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r119:/Table/74/1/{3/5-5} [(n3,s3):1, (n2,s2):2, next=3, gen=83] I190903 20:13:08.295945 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:08.306387 82029 storage/replica_raft.go:291 [n3,s3,r119/1:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:08.306433 82029 storage/replica_raft.go:304 [n3,s3,r119/1:/Table/74/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.307313 82029 storage/replica_command.go:979 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r119 failed: storage/replica_raft.go:305: [n3,s3,r119/1:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:08.307345 82029 storage/queue.go:1027 [n3,replicate,s3,r119/1:/Table/74/1/{3/5-5}] change replicas of r119 failed: storage/replica_raft.go:305: [n3,s3,r119/1:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.307649 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.308512 82518 storage/replica_command.go:1430 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r121:/Table/74/1/5{-/7} [(n3,s3):1, next=2, gen=82] I190903 20:13:08.314612 82518 storage/replica_raft.go:291 [n3,s3,r121/1:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:08.315709 82518 storage/store_snapshot.go:995 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] sending LEARNER snapshot 7400259c at applied index 18 I190903 20:13:08.315858 82518 storage/store_snapshot.go:1038 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.316831 82371 storage/replica_raftstorage.go:793 [n1,s1,r121/2:{-}] applying LEARNER snapshot [id=7400259c index=18] I190903 20:13:08.318103 82371 storage/replica_raftstorage.go:814 [n1,s1,r121/2:/Table/74/1/5{-/7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=7400259c index=18] I190903 20:13:08.319096 82518 storage/replica_command.go:1430 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r121:/Table/74/1/5{-/7} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=83] I190903 20:13:08.327823 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:08.328588 82518 storage/replica_raft.go:291 [n3,s3,r121/1:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:08.329158 82518 storage/replica_command.go:1430 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r121:/Table/74/1/5{-/7} [(n3,s3):1, (n1,s1):2, next=3, gen=84] I190903 20:13:08.343479 82518 storage/replica_raft.go:291 [n3,s3,r121/1:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:08.343523 82518 storage/replica_raft.go:304 [n3,s3,r121/1:/Table/74/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.344337 82518 storage/replica_command.go:979 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r121 failed: storage/replica_raft.go:305: [n3,s3,r121/1:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:08.344378 82518 storage/queue.go:1027 [n3,replicate,s3,r121/1:/Table/74/1/5{-/7}] change replicas of r121 failed: storage/replica_raft.go:305: [n3,s3,r121/1:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.344784 82593 storage/replica_command.go:1430 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r122:/Table/74/1/5/{7-8} [(n3,s3):1, next=2, gen=83] I190903 20:13:08.351311 82593 storage/replica_raft.go:291 [n3,s3,r122/1:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:08.352449 82593 storage/store_snapshot.go:995 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] sending LEARNER snapshot 3d76bd52 at applied index 18 I190903 20:13:08.352636 82593 storage/store_snapshot.go:1038 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.353968 82730 storage/replica_raftstorage.go:793 [n2,s2,r122/2:{-}] applying LEARNER snapshot [id=3d76bd52 index=18] I190903 20:13:08.355109 82730 storage/replica_raftstorage.go:814 [n2,s2,r122/2:/Table/74/1/5/{7-8}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=3d76bd52 index=18] I190903 20:13:08.355795 82593 storage/replica_command.go:1430 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] change replicas (add [(n2,s2):2] remove []): existing descriptor r122:/Table/74/1/5/{7-8} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=84] I190903 20:13:08.362360 82593 storage/replica_raft.go:291 [n3,s3,r122/1:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:08.362834 82593 storage/replica_command.go:1430 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r122:/Table/74/1/5/{7-8} [(n3,s3):1, (n2,s2):2, next=3, gen=85] I190903 20:13:08.365101 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.374707 82593 storage/replica_raft.go:291 [n3,s3,r122/1:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:08.374753 82593 storage/replica_raft.go:304 [n3,s3,r122/1:/Table/74/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.375623 82593 storage/replica_command.go:979 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r122 failed: storage/replica_raft.go:305: [n3,s3,r122/1:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:08.375656 82593 storage/queue.go:1027 [n3,replicate,s3,r122/1:/Table/74/1/5/{7-8}] change replicas of r122 failed: storage/replica_raft.go:305: [n3,s3,r122/1:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.376233 82815 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [(n1,s1):6LEARNER] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):5, next=6, gen=12] I190903 20:13:08.383739 82815 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing ADD_REPLICA[(n1,s1):6LEARNER]: after=[(n3,s3):5 (n1,s1):6LEARNER] next=7 I190903 20:13:08.385334 82815 storage/store_snapshot.go:995 [n3,replicate,s3,r4/5:/System{/tsd-tse}] sending LEARNER snapshot 044df35a at applied index 108 I190903 20:13:08.389770 82815 storage/store_snapshot.go:1038 [n3,replicate,s3,r4/5:/System{/tsd-tse}] streamed snapshot to (n1,s1):6LEARNER: kv pairs: 1962, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:08.397287 82860 storage/replica_raftstorage.go:793 [n1,s1,r4/6:{-}] applying LEARNER snapshot [id=044df35a index=108] I190903 20:13:08.398897 82860 storage/replica_raftstorage.go:814 [n1,s1,r4/6:/System{/tsd-tse}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=044df35a index=108] I190903 20:13:08.399845 82815 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [(n1,s1):6] remove []): existing descriptor r4:/System{/tsd-tse} [(n3,s3):5, (n1,s1):6LEARNER, next=7, gen=13] I190903 20:13:08.410297 82815 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing ADD_REPLICA[(n1,s1):6]: after=[(n3,s3):5 (n1,s1):6] next=7 I190903 20:13:08.410908 82815 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [] remove [(n3,s3):5]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):5, (n1,s1):6, next=7, gen=14] I190903 20:13:08.419449 82815 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n3,s3):5]: after=[(n1,s1):6] next=7 E190903 20:13:08.419494 82815 storage/replica_raft.go:304 [n3,s3,r4/5:/System{/tsd-tse}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):5]: after=[(n1,s1):6] next=7 to remove self (leaseholder) I190903 20:13:08.420367 82815 storage/replica_command.go:979 [n3,replicate,s3,r4/5:/System{/tsd-tse}] could not promote [] to voter, rolling back: change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/5:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):5]: after=[(n1,s1):6] next=7 to remove self (leaseholder) E190903 20:13:08.420430 82815 storage/queue.go:1027 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas of r4 failed: storage/replica_raft.go:305: [n3,s3,r4/5:/System{/tsd-tse}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):5]: after=[(n1,s1):6] next=7 to remove self (leaseholder) I190903 20:13:08.420859 82827 storage/replica_command.go:1430 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r124:/Table/74/1/{5/9-6} [(n3,s3):1, next=2, gen=85] I190903 20:13:08.425118 1771 storage/compactor/compactor.go:370 [n2,s2,compactor] processing compaction #1/17 (/System/tsd-/System/"tse") for 464 KiB (reasons: size=false used=true avail=false) I190903 20:13:08.428835 82827 storage/replica_raft.go:291 [n3,s3,r124/1:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:08.430854 82827 storage/store_snapshot.go:995 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] sending LEARNER snapshot b0b263d5 at applied index 18 I190903 20:13:08.431013 82827 storage/store_snapshot.go:1038 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.431967 82792 storage/replica_raftstorage.go:793 [n1,s1,r124/2:{-}] applying LEARNER snapshot [id=b0b263d5 index=18] I190903 20:13:08.437552 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:08.441527 82792 storage/replica_raftstorage.go:814 [n1,s1,r124/2:/Table/74/1/{5/9-6}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=b0b263d5 index=18] I190903 20:13:08.442318 82827 storage/replica_command.go:1430 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r124:/Table/74/1/{5/9-6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=86] I190903 20:13:08.455088 1771 storage/compactor/compactor.go:386 [n2,s2,compactor] processed compaction #1/17 (/System/tsd-/System/"tse") for 464 KiB in 0.0s I190903 20:13:08.455426 82827 storage/replica_raft.go:291 [n3,s3,r124/1:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:08.456209 82827 storage/replica_command.go:1430 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r124:/Table/74/1/{5/9-6} [(n3,s3):1, (n1,s1):2, next=3, gen=87] I190903 20:13:08.489357 82827 storage/replica_raft.go:291 [n3,s3,r124/1:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:08.489408 82827 storage/replica_raft.go:304 [n3,s3,r124/1:/Table/74/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.490331 82827 storage/replica_command.go:979 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r124 failed: storage/replica_raft.go:305: [n3,s3,r124/1:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:08.490374 82827 storage/queue.go:1027 [n3,replicate,s3,r124/1:/Table/74/1/{5/9-6}] change replicas of r124 failed: storage/replica_raft.go:305: [n3,s3,r124/1:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.490877 83197 storage/replica_command.go:1430 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r42:/Table/6{4/2-6} [(n3,s3):2, next=3, gen=38] I190903 20:13:08.507408 83197 storage/replica_raft.go:291 [n3,s3,r42/2:/Table/6{4/2-6}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:13:08.509198 83197 storage/store_snapshot.go:995 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] sending LEARNER snapshot 534da48b at applied index 38 I190903 20:13:08.509397 83197 storage/store_snapshot.go:1038 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.510832 83349 storage/replica_raftstorage.go:793 [n2,s2,r42/3:{-}] applying LEARNER snapshot [id=534da48b index=38] I190903 20:13:08.512142 83349 storage/replica_raftstorage.go:814 [n2,s2,r42/3:/Table/6{4/2-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=534da48b index=38] I190903 20:13:08.512638 83197 storage/replica_command.go:1430 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] change replicas (add [(n2,s2):3] remove []): existing descriptor r42:/Table/6{4/2-6} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=39] I190903 20:13:08.523456 83197 storage/replica_raft.go:291 [n3,s3,r42/2:/Table/6{4/2-6}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 20:13:08.524743 83197 storage/replica_command.go:1430 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r42:/Table/6{4/2-6} [(n3,s3):2, (n2,s2):3, next=4, gen=40] I190903 20:13:08.540449 83197 storage/replica_raft.go:291 [n3,s3,r42/2:/Table/6{4/2-6}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 E190903 20:13:08.540494 83197 storage/replica_raft.go:304 [n3,s3,r42/2:/Table/6{4/2-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:08.541307 83197 storage/replica_command.go:979 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] could not promote [] to voter, rolling back: change replicas of r42 failed: storage/replica_raft.go:305: [n3,s3,r42/2:/Table/6{4/2-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:08.541368 83197 storage/queue.go:1027 [n3,replicate,s3,r42/2:/Table/6{4/2-6}] change replicas of r42 failed: storage/replica_raft.go:305: [n3,s3,r42/2:/Table/6{4/2-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:08.543602 83038 storage/replica_command.go:1430 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r125:/Table/74/{1/6-2} [(n3,s3):1, next=2, gen=86] E190903 20:13:08.543671 83454 storage/queue.go:1027 [n2,replicate,s2,r120/2:/Table/74/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 20:13:08.554631 83038 storage/replica_raft.go:291 [n3,s3,r125/1:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:08.555784 83038 storage/store_snapshot.go:995 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] sending LEARNER snapshot d7c323d3 at applied index 19 I190903 20:13:08.555926 83038 storage/store_snapshot.go:1038 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.557073 83509 storage/replica_raftstorage.go:793 [n2,s2,r125/2:{-}] applying LEARNER snapshot [id=d7c323d3 index=19] I190903 20:13:08.559012 83509 storage/replica_raftstorage.go:814 [n2,s2,r125/2:/Table/74/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d7c323d3 index=19] I190903 20:13:08.559802 83038 storage/replica_command.go:1430 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r125:/Table/74/{1/6-2} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=87] I190903 20:13:08.567631 83038 storage/replica_raft.go:291 [n3,s3,r125/1:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:08.568148 83038 storage/replica_command.go:1430 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r125:/Table/74/{1/6-2} [(n3,s3):1, (n2,s2):2, next=3, gen=88] I190903 20:13:08.575931 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:08.581626 83038 storage/replica_raft.go:291 [n3,s3,r125/1:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:08.581675 83038 storage/replica_raft.go:304 [n3,s3,r125/1:/Table/74/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.582807 83038 storage/replica_command.go:979 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r125 failed: storage/replica_raft.go:305: [n3,s3,r125/1:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:08.582855 83038 storage/queue.go:1027 [n3,replicate,s3,r125/1:/Table/74/{1/6-2}] change replicas of r125 failed: storage/replica_raft.go:305: [n3,s3,r125/1:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:08.585534 83669 storage/replica_command.go:1430 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r126:/{Table/74/2-Max} [(n3,s3):1, next=2, gen=86] E190903 20:13:08.589966 83684 storage/queue.go:1027 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:08.601082 83669 storage/replica_raft.go:291 [n3,s3,r126/1:/{Table/74/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:08.602302 83669 storage/store_snapshot.go:995 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] sending LEARNER snapshot aa6691bf at applied index 15 I190903 20:13:08.602468 83669 storage/store_snapshot.go:1038 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 6, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:08.604802 83633 storage/replica_raftstorage.go:793 [n1,s1,r126/2:{-}] applying LEARNER snapshot [id=aa6691bf index=15] I190903 20:13:08.607803 83633 storage/replica_raftstorage.go:814 [n1,s1,r126/2:/{Table/74/2-Max}] applied LEARNER snapshot [total=3ms ingestion=4@3ms id=aa6691bf index=15] I190903 20:13:08.608349 83669 storage/replica_command.go:1430 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r126:/{Table/74/2-Max} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=87] I190903 20:13:08.621227 83669 storage/replica_raft.go:291 [n3,s3,r126/1:/{Table/74/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:08.622011 83669 storage/replica_command.go:1430 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r126:/{Table/74/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=88] I190903 20:13:08.635617 83669 storage/replica_raft.go:291 [n3,s3,r126/1:/{Table/74/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:08.635664 83669 storage/replica_raft.go:304 [n3,s3,r126/1:/{Table/74/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:08.636492 83669 storage/replica_command.go:979 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] could not promote [] to voter, rolling back: change replicas of r126 failed: storage/replica_raft.go:305: [n3,s3,r126/1:/{Table/74/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:08.636532 83669 storage/queue.go:1027 [n3,replicate,s3,r126/1:/{Table/74/2-Max}] change replicas of r126 failed: storage/replica_raft.go:305: [n3,s3,r126/1:/{Table/74/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:08.640763 83848 storage/queue.go:1027 [n1,replicate,s1,r126/2:/{Table/74/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:21] E190903 20:13:08.640909 83908 storage/queue.go:1027 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:08.640916 83901 storage/replica_command.go:1430 [n2,replicate,s2,r120/2:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r120:/Table/74/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=82] I190903 20:13:08.650113 83901 storage/replica_raft.go:291 [n2,s2,r120/2:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:08.659267 83819 storage/store.go:2593 [n3,replicaGC,s3,r120/1:/Table/74/1{-/3/4}] removing replica r120/1 I190903 20:13:08.659684 83819 storage/replica_destroy.go:152 [n3,replicaGC,s3,r120/1:/Table/74/1{-/3/4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:08.677983 84027 storage/replica_command.go:1430 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r118:/Table/74/1/3/{4-5} [(n3,s3):1, (n1,s1):2, next=3, gen=82] I190903 20:13:08.680876 84027 storage/replica_raft.go:291 [n1,s1,r118/2:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:08.689922 84030 storage/store.go:2593 [n3,replicaGC,s3,r118/1:/Table/74/1/3/{4-5}] removing replica r118/1 I190903 20:13:08.690221 84030 storage/replica_destroy.go:152 [n3,replicaGC,s3,r118/1:/Table/74/1/3/{4-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:08.736407 84106 storage/queue.go:1027 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:08.788961 84333 storage/queue.go:1027 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:08.801161 84372 storage/queue.go:1027 [n2,replicate,s2,r42/3:/Table/6{4/2-6}] no removable replicas from range that needs a removal: [2:0, 3*:45] E190903 20:13:08.818981 84455 storage/queue.go:1027 [n1,replicate,s1,r4/6:/System{/tsd-tse}] no removable replicas from range that needs a removal: [5:0, 6*:115] I190903 20:13:08.827169 84447 storage/replica_command.go:1430 [n1,replicate,s1,r126/2:/{Table/74/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r126:/{Table/74/2-Max} [(n3,s3):1, (n1,s1):2, next=3, gen=88] I190903 20:13:08.830203 84447 storage/replica_raft.go:291 [n1,s1,r126/2:/{Table/74/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:08.838198 84458 storage/store.go:2593 [n3,replicaGC,s3,r126/1:/{Table/74/2-Max}] removing replica r126/1 I190903 20:13:08.838607 84458 storage/replica_destroy.go:152 [n3,replicaGC,s3,r126/1:/{Table/74/2-Max}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:08.850797 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:08.852968 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:08.879550 84049 storage/replica_command.go:1430 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r121:/Table/74/1/5{-/7} [(n3,s3):1, (n1,s1):2, next=3, gen=84] I190903 20:13:08.883714 84049 storage/replica_raft.go:291 [n1,s1,r121/2:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:08.891411 84677 storage/store.go:2593 [n3,replicaGC,s3,r121/1:/Table/74/1/5{-/7}] removing replica r121/1 I190903 20:13:08.891793 84677 storage/replica_destroy.go:152 [n3,replicaGC,s3,r121/1:/Table/74/1/5{-/7}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:08.930864 84511 storage/replica_command.go:1430 [n2,replicate,s2,r42/3:/Table/6{4/2-6}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r42:/Table/6{4/2-6} [(n3,s3):2, (n2,s2):3, next=4, gen=40] I190903 20:13:08.937159 84511 storage/replica_raft.go:291 [n2,s2,r42/3:/Table/6{4/2-6}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 20:13:08.938264 84801 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r119:/Table/74/1/{3/5-5} [(n3,s3):1, (n2,s2):2, next=3, gen=83] I190903 20:13:08.948583 84885 storage/store.go:2593 [n3,replicaGC,s3,r42/2:/Table/6{4/2-6}] removing replica r42/2 I190903 20:13:08.950119 84885 storage/replica_destroy.go:152 [n3,replicaGC,s3,r42/2:/Table/6{4/2-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:08.955323 84801 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:08.969636 83721 storage/store.go:2593 [n3,replicaGC,s3,r119/1:/Table/74/1/{3/5-5}] removing replica r119/1 I190903 20:13:08.970021 83721 storage/replica_destroy.go:152 [n3,replicaGC,s3,r119/1:/Table/74/1/{3/5-5}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:09.062268 85010 storage/replica_command.go:1430 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r124:/Table/74/1/{5/9-6} [(n3,s3):1, (n1,s1):2, next=3, gen=87] I190903 20:13:09.065268 85010 storage/replica_raft.go:291 [n1,s1,r124/2:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:09.073130 84643 storage/store.go:2593 [n3,replicaGC,s3,r124/1:/Table/74/1/{5/9-6}] removing replica r124/1 I190903 20:13:09.073515 84643 storage/replica_destroy.go:152 [n3,replicaGC,s3,r124/1:/Table/74/1/{5/9-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:09.153239 85352 storage/queue.go:1027 [n3,replicate,s3,r4/5:/System{/tsd-tse}] no removable replicas from range that needs a removal: [5*:119, 6:0] I190903 20:13:09.270650 85776 storage/replica_command.go:1430 [n3,replicate,s3,r4/5:/System{/tsd-tse}] change replicas (add [] remove [(n1,s1):6]): existing descriptor r4:/System{/tsd-tse} [(n3,s3):5, (n1,s1):6, next=7, gen=14] I190903 20:13:09.279573 85776 storage/replica_raft.go:291 [n3,s3,r4/5:/System{/tsd-tse}] proposing REMOVE_REPLICA[(n1,s1):6]: after=[(n3,s3):5] next=7 I190903 20:13:09.298441 85738 storage/store.go:2593 [n1,replicaGC,s1,r4/6:/System{/tsd-tse}] removing replica r4/6 I190903 20:13:09.298901 85738 storage/replica_destroy.go:152 [n1,replicaGC,s1,r4/6:/System{/tsd-tse}] removed 1947 (1941+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:09.321424 85889 storage/replica_command.go:1430 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r122:/Table/74/1/5/{7-8} [(n3,s3):1, (n2,s2):2, next=3, gen=85] I190903 20:13:09.334387 85889 storage/replica_raft.go:291 [n2,s2,r122/2:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:09.335776 85911 storage/replica_command.go:1430 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r125:/Table/74/{1/6-2} [(n3,s3):1, (n2,s2):2, next=3, gen=88] I190903 20:13:09.345298 85973 storage/store.go:2593 [n3,replicaGC,s3,r122/1:/Table/74/1/5/{7-8}] removing replica r122/1 I190903 20:13:09.345700 85973 storage/replica_destroy.go:152 [n3,replicaGC,s3,r122/1:/Table/74/1/5/{7-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:09.346144 85911 storage/replica_raft.go:291 [n2,s2,r125/2:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:09.359153 82209 storage/store.go:2593 [n3,replicaGC,s3,r125/1:/Table/74/{1/6-2}] removing replica r125/1 I190903 20:13:09.359506 82209 storage/replica_destroy.go:152 [n3,replicaGC,s3,r125/1:/Table/74/{1/6-2}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:09.448094 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:09.454468 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:09.458570 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:09.460440 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a > 5 I190903 20:13:09.467346 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:09.471244 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:09.473080 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:09.484126 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:09.504516 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=2 mtime=2019-09-03 20:13:09.502689275 +0000 UTC I190903 20:13:09.512575 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 1}] I190903 20:13:09.519207 86449 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=2:1567541901.500247179,0, refcount=0 I190903 20:13:09.563887 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=74 (multi col list partitioning - DEFAULT) version=3 mtime=2019-09-03 20:13:09.563657465 +0000 UTC I190903 20:13:09.569616 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 2}] I190903 20:13:09.570392 86948 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=3:1567541901.501247179,0, refcount=0 I190903 20:13:09.625061 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=4 mtime=2019-09-03 20:13:09.60520117 +0000 UTC I190903 20:13:09.643100 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 74, info: {TableName:data.public."multi col list partitioning - DEFAULT" Statement:ALTER TABLE "multi col list partitioning - DEFAULT" PARTITION BY LIST (a, b) (PARTITION p34 VALUES IN ((3, 4)), PARTITION p57 VALUES IN ((5, 7)), PARTITION p58 VALUES IN ((5, 8)), PARTITION p5d VALUES IN ((5, DEFAULT))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:09.648119 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:09.648174 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:09.648367 87412 sql/lease.go:923 new lease: 74("multi col list partitioning - DEFAULT") ver=4:1567541901.502247179,0, refcount=0 I190903 20:13:09.649587 87637 storage/replica_command.go:1430 [n2,s2,r120/2:/Table/74/1{-/3/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r120:/Table/74/1{-/3/4} [(n2,s2):2, next=3, gen=83] I190903 20:13:09.650262 87616 storage/replica_command.go:1430 [n1,s1,r126/2:/{Table/74/2-Max}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r126:/{Table/74/2-Max} [(n1,s1):2, next=3, gen=89] I190903 20:13:09.653452 87616 storage/replica_raft.go:291 [n1,s1,r126/2:/{Table/74/2-Max}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:09.654727 87616 storage/store_snapshot.go:995 [n1,s1,r126/2:/{Table/74/2-Max}] sending LEARNER snapshot 0d3047e4 at applied index 27 I190903 20:13:09.654893 87616 storage/store_snapshot.go:1038 [n1,s1,r126/2:/{Table/74/2-Max}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:09.655717 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 3}] I190903 20:13:09.656145 87688 storage/replica_raftstorage.go:793 [n2,s2,r126/3:{-}] applying LEARNER snapshot [id=0d3047e4 index=27] I190903 20:13:09.657418 87688 storage/replica_raftstorage.go:814 [n2,s2,r126/3:/{Table/74/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=0d3047e4 index=27] I190903 20:13:09.658202 87616 storage/replica_command.go:1430 [n1,s1,r126/2:/{Table/74/2-Max}] change replicas (add [(n2,s2):3] remove []): existing descriptor r126:/{Table/74/2-Max} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=90] I190903 20:13:09.659510 87637 storage/replica_raft.go:291 [n2,s2,r120/2:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:13:09.661002 87637 storage/store_snapshot.go:995 [n2,s2,r120/2:/Table/74/1{-/3/4}] sending LEARNER snapshot 66bab3a1 at applied index 31 I190903 20:13:09.661191 87637 storage/store_snapshot.go:1038 [n2,s2,r120/2:/Table/74/1{-/3/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:09.661583 87779 storage/raft_snapshot_queue.go:120 [n2,raftsnapshot,s2,r120/2:/Table/74/1{-/3/4}] skipping snapshot; replica is likely a learner in the process of being added: (n3,s3):3LEARNER I190903 20:13:09.663247 87777 storage/replica_raftstorage.go:793 [n3,s3,r120/3:{-}] applying LEARNER snapshot [id=66bab3a1 index=31] I190903 20:13:09.668781 87777 storage/replica_raftstorage.go:814 [n3,s3,r120/3:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=5ms ingestion=4@5ms id=66bab3a1 index=31] I190903 20:13:09.669465 87637 storage/replica_command.go:1430 [n2,s2,r120/2:/Table/74/1{-/3/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r120:/Table/74/1{-/3/4} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=84] I190903 20:13:09.669630 87616 storage/replica_raft.go:291 [n1,s1,r126/2:/{Table/74/2-Max}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 E190903 20:13:09.672010 87243 storage/queue.go:1027 [n2,replicate,s2,r126/3:/{Table/74/2-Max}] no removable replicas from range that needs a removal: [2:0, 3*:32] I190903 20:13:09.682118 87637 storage/replica_raft.go:291 [n2,s2,r120/2:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 E190903 20:13:09.691404 87650 storage/queue.go:1027 [n3,replicate,s3,r120/3:/Table/74/1{-/3/4}] no removable replicas from range that needs a removal: [2:0, 3*:36] I190903 20:13:09.693221 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:09.715233 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p34 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:09.748621 88520 storage/replica_command.go:1430 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r118:/Table/74/1/3/{4-5} [(n1,s1):2, next=3, gen=83] I190903 20:13:09.784809 88520 storage/replica_raft.go:291 [n1,s1,r118/2:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:09.784947 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p57 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:09.785949 88520 storage/store_snapshot.go:995 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] sending LEARNER snapshot 1f220ebe at applied index 33 I190903 20:13:09.786114 88520 storage/store_snapshot.go:1038 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:09.787760 88561 storage/replica_raftstorage.go:793 [n2,s2,r118/3:{-}] applying LEARNER snapshot [id=1f220ebe index=33] I190903 20:13:09.801712 88561 storage/replica_raftstorage.go:814 [n2,s2,r118/3:/Table/74/1/3/{4-5}] applied LEARNER snapshot [total=14ms ingestion=4@1ms id=1f220ebe index=33] I190903 20:13:09.802517 88520 storage/replica_command.go:1430 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] change replicas (add [(n2,s2):3] remove []): existing descriptor r118:/Table/74/1/3/{4-5} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=84] I190903 20:13:09.806461 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p58 OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:09.810579 88524 storage/replica_command.go:1430 [n2,s2,r126/3:/{Table/74/2-Max}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r126:/{Table/74/2-Max} [(n1,s1):2, (n2,s2):3, next=4, gen=91] I190903 20:13:09.812190 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:09.831400 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 74, info: {Target:PARTITION p5d OF INDEX data.public."multi col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:09.832328 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:09.836033 88520 storage/replica_raft.go:291 [n1,s1,r118/2:/Table/74/1/3/{4-5}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:09.838519 88520 storage/replica_command.go:1430 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r118:/Table/74/1/3/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=85] I190903 20:13:09.844498 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:09.849900 88828 storage/replica_command.go:1430 [n3,s3,r120/3:/Table/74/1{-/3/4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r120:/Table/74/1{-/3/4} [(n2,s2):2, (n3,s3):3, next=4, gen=85] I190903 20:13:09.857633 89616 storage/replica_command.go:1430 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r122:/Table/74/1/5/{7-8} [(n2,s2):2, next=3, gen=86] I190903 20:13:09.859705 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:09.865184 88828 storage/replica_raft.go:291 [n3,s3,r120/3:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 20:13:09.868123 87553 storage/replica_command.go:609 [n3,merge,s3,r117/1:/Table/74{-/1}] initiating a merge of r120:/Table/74/1{-/3/4} [(n3,s3):3, next=4, gen=86] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:09.883374 89616 storage/replica_raft.go:291 [n2,s2,r122/2:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:13:09.885428 89616 storage/store_snapshot.go:995 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] sending LEARNER snapshot 3f2087d7 at applied index 35 I190903 20:13:09.885618 89616 storage/store_snapshot.go:1038 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:09.885903 88524 storage/replica_raft.go:291 [n2,s2,r126/3:/{Table/74/2-Max}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:09.887171 89708 storage/replica_raftstorage.go:793 [n3,s3,r122/3:{-}] applying LEARNER snapshot [id=3f2087d7 index=35] I190903 20:13:09.888016 87605 storage/replica_command.go:609 [n2,merge,s2,r125/2:/Table/74/{1/6-2}] initiating a merge of r126:/{Table/74/2-Max} [(n2,s2):3, next=4, gen=92] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:09.897867 89730 storage/store.go:2593 [n1,replicaGC,s1,r126/2:/{Table/74/2-Max}] removing replica r126/2 I190903 20:13:09.900349 89708 storage/replica_raftstorage.go:814 [n3,s3,r122/3:/Table/74/1/5/{7-8}] applied LEARNER snapshot [total=13ms ingestion=4@13ms id=3f2087d7 index=35] I190903 20:13:09.900840 89730 storage/replica_destroy.go:152 [n1,replicaGC,s1,r126/2:/{Table/74/2-Max}] removed 4 (0+4) keys in 2ms [clear=0ms commit=2ms] I190903 20:13:09.906164 89616 storage/replica_command.go:1430 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] change replicas (add [(n3,s3):3] remove []): existing descriptor r122:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=87] I190903 20:13:09.906236 2273 storage/store.go:2593 [n3,s3,r117/1:/Table/74{-/1}] removing replica r120/3 I190903 20:13:09.926336 89616 storage/replica_raft.go:291 [n2,s2,r122/2:/Table/74/1/5/{7-8}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 20:13:09.927925 89616 storage/replica_command.go:1430 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r122:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=88] I190903 20:13:09.935692 87605 storage/replica_command.go:609 [n2,merge,s2,r125/2:/Table/74/{1/6-2}] initiating a merge of r126:/{Table/74/2-Max} [(n2,s2):3, next=4, gen=92] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:09.939937 89616 storage/replica_raft.go:291 [n2,s2,r122/2:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:13:09.939986 89616 storage/replica_raft.go:304 [n2,s2,r122/2:/Table/74/1/5/{7-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:09.945325 89616 storage/replica_command.go:979 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] could not promote [] to voter, rolling back: change replicas of r122 failed: storage/replica_raft.go:305: [n2,s2,r122/2:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:09.945402 89616 storage/queue.go:1027 [n2,replicate,s2,r122/2:/Table/74/1/5/{7-8}] change replicas of r122 failed: storage/replica_raft.go:305: [n2,s2,r122/2:/Table/74/1/5/{7-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:09.953459 1641 storage/store.go:2593 [n2,s2,r125/2:/Table/74/{1/6-2}] removing replica r126/3 I190903 20:13:10.030829 90236 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r119:/Table/74/1/{3/5-5} [(n2,s2):2, next=3, gen=84] I190903 20:13:10.038017 154 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:10.039952 90236 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:10.043649 90236 storage/store_snapshot.go:995 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] sending LEARNER snapshot 29809cfb at applied index 36 I190903 20:13:10.051334 90236 storage/store_snapshot.go:1038 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:10.052455 90278 storage/replica_raftstorage.go:793 [n1,s1,r119/3:{-}] applying LEARNER snapshot [id=29809cfb index=36] I190903 20:13:10.053821 90278 storage/replica_raftstorage.go:814 [n1,s1,r119/3:/Table/74/1/{3/5-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=29809cfb index=36] I190903 20:13:10.054489 90236 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r119:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=85] I190903 20:13:10.074088 90236 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/74/1/{3/5-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:10.074656 90236 storage/replica_command.go:1430 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r119:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3, next=4, gen=86] I190903 20:13:10.085260 90442 storage/replica_command.go:284 [n2,split,s2,r125/2:/{Table/74/1/6-Max}] initiating a split of this range at key /Table/74/2 [r57] (zone config) I190903 20:13:10.087888 90236 storage/replica_raft.go:291 [n2,s2,r119/2:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:10.087922 90236 storage/replica_raft.go:304 [n2,s2,r119/2:/Table/74/1/{3/5-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:10.088768 90236 storage/replica_command.go:979 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] could not promote [] to voter, rolling back: change replicas of r119 failed: storage/replica_raft.go:305: [n2,s2,r119/2:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:10.088811 90236 storage/queue.go:1027 [n2,replicate,s2,r119/2:/Table/74/1/{3/5-5}] change replicas of r119 failed: storage/replica_raft.go:305: [n2,s2,r119/2:/Table/74/1/{3/5-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:10.148775 90634 storage/replica_command.go:1430 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r125:/Table/74/{1/6-2} [(n2,s2):2, next=3, gen=94] I190903 20:13:10.163035 90634 storage/replica_raft.go:291 [n2,s2,r125/2:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:10.164142 90634 storage/store_snapshot.go:995 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] sending LEARNER snapshot 1117d593 at applied index 45 I190903 20:13:10.164376 90634 storage/store_snapshot.go:1038 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 18, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.165715 90683 storage/replica_raftstorage.go:793 [n1,s1,r125/3:{-}] applying LEARNER snapshot [id=1117d593 index=45] I190903 20:13:10.167199 90683 storage/replica_raftstorage.go:814 [n1,s1,r125/3:/Table/74/{1/6-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=1117d593 index=45] I190903 20:13:10.167653 90634 storage/replica_command.go:1430 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r125:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=95] I190903 20:13:10.176248 90634 storage/replica_raft.go:291 [n2,s2,r125/2:/Table/74/{1/6-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:10.176803 90634 storage/replica_command.go:1430 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r125:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3, next=4, gen=96] I190903 20:13:10.191643 90634 storage/replica_raft.go:291 [n2,s2,r125/2:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:10.191740 90634 storage/replica_raft.go:304 [n2,s2,r125/2:/Table/74/{1/6-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:10.192859 90634 storage/replica_command.go:979 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] could not promote [] to voter, rolling back: change replicas of r125 failed: storage/replica_raft.go:305: [n2,s2,r125/2:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:10.192907 90634 storage/queue.go:1027 [n2,replicate,s2,r125/2:/Table/74/{1/6-2}] change replicas of r125 failed: storage/replica_raft.go:305: [n2,s2,r125/2:/Table/74/{1/6-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:10.198232 90810 storage/replica_command.go:1430 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r123:/Table/74/1/5/{8-9} [(n3,s3):1, next=2, gen=84] I190903 20:13:10.205474 90810 storage/replica_raft.go:291 [n3,s3,r123/1:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:10.206909 90810 storage/store_snapshot.go:995 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] sending LEARNER snapshot 5b15258a at applied index 20 I190903 20:13:10.207065 90810 storage/store_snapshot.go:1038 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.208521 89817 storage/replica_raftstorage.go:793 [n1,s1,r123/2:{-}] applying LEARNER snapshot [id=5b15258a index=20] I190903 20:13:10.209826 89817 storage/replica_raftstorage.go:814 [n1,s1,r123/2:/Table/74/1/5/{8-9}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5b15258a index=20] I190903 20:13:10.210266 90810 storage/replica_command.go:1430 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] change replicas (add [(n1,s1):2] remove []): existing descriptor r123:/Table/74/1/5/{8-9} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=85] I190903 20:13:10.217470 90810 storage/replica_raft.go:291 [n3,s3,r123/1:/Table/74/1/5/{8-9}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:10.217987 90810 storage/replica_command.go:1430 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r123:/Table/74/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=86] I190903 20:13:10.234610 90810 storage/replica_raft.go:291 [n3,s3,r123/1:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:10.234761 90810 storage/replica_raft.go:304 [n3,s3,r123/1:/Table/74/1/5/{8-9}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:10.235511 90810 storage/replica_command.go:979 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] could not promote [] to voter, rolling back: change replicas of r123 failed: storage/replica_raft.go:305: [n3,s3,r123/1:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:10.235555 90810 storage/queue.go:1027 [n3,replicate,s3,r123/1:/Table/74/1/5/{8-9}] change replicas of r123 failed: storage/replica_raft.go:305: [n3,s3,r123/1:/Table/74/1/5/{8-9}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:10.252736 91007 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, next=4, gen=27] I190903 20:13:10.261231 91007 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n2,s2):2 (n3,s3):4LEARNER] next=5 I190903 20:13:10.262416 91007 storage/store_snapshot.go:995 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] sending LEARNER snapshot 078dfc4b at applied index 43 I190903 20:13:10.262567 91007 storage/store_snapshot.go:1038 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.263706 90941 storage/replica_raftstorage.go:793 [n3,s3,r39/4:{-}] applying LEARNER snapshot [id=078dfc4b index=43] I190903 20:13:10.265006 90941 storage/replica_raftstorage.go:814 [n3,s3,r39/4:/Table/6{0/2-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=078dfc4b index=43] I190903 20:13:10.265522 91007 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [(n3,s3):4] remove []): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n3,s3):4LEARNER, next=5, gen=28] I190903 20:13:10.284575 91007 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n2,s2):2 (n3,s3):4] next=5 I190903 20:13:10.285116 91007 storage/replica_command.go:1430 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n3,s3):4, next=5, gen=29] I190903 20:13:10.296196 91007 storage/replica_raft.go:291 [n2,s2,r39/2:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):4] next=5 E190903 20:13:10.296240 91007 storage/replica_raft.go:304 [n2,s2,r39/2:/Table/6{0/2-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:13:10.297197 91007 storage/replica_command.go:979 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] could not promote [] to voter, rolling back: change replicas of r39 failed: storage/replica_raft.go:305: [n2,s2,r39/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 20:13:10.297243 91007 storage/queue.go:1027 [n2,replicate,s2,r39/2:/Table/6{0/2-2}] change replicas of r39 failed: storage/replica_raft.go:305: [n2,s2,r39/2:/Table/6{0/2-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:13:10.352721 91348 storage/replica_command.go:284 [n3,split,s3,r117/1:/Table/74{-/1/3/4}] initiating a split of this range at key /Table/74/1 [r127] (zone config) E190903 20:13:10.373333 91336 storage/queue.go:1027 [n3,replicate,s3,r122/3:/Table/74/1/5/{7-8}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 20:13:10.418788 91480 storage/replica_command.go:1430 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r127:/Table/74/1{-/3/4} [(n3,s3):1, next=2, gen=88] I190903 20:13:10.443816 91480 storage/replica_raft.go:291 [n3,s3,r127/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:10.445011 91480 storage/store_snapshot.go:995 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] sending LEARNER snapshot 3128a7ad at applied index 14 I190903 20:13:10.445198 91480 storage/store_snapshot.go:1038 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 18, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.448186 91490 storage/replica_raftstorage.go:793 [n1,s1,r127/2:{-}] applying LEARNER snapshot [id=3128a7ad index=14] I190903 20:13:10.465247 91490 storage/replica_raftstorage.go:814 [n1,s1,r127/2:/Table/74/1{-/3/4}] applied LEARNER snapshot [total=17ms ingestion=4@9ms id=3128a7ad index=14] I190903 20:13:10.468893 91480 storage/replica_command.go:1430 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r127:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=89] I190903 20:13:10.482889 91480 storage/replica_raft.go:291 [n3,s3,r127/1:/Table/74/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:10.483435 91480 storage/replica_command.go:1430 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r127:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=90] I190903 20:13:10.550924 91480 storage/replica_raft.go:291 [n3,s3,r127/1:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:10.550977 91480 storage/replica_raft.go:304 [n3,s3,r127/1:/Table/74/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:10.553760 91480 storage/replica_command.go:979 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:10.553789 91480 storage/queue.go:1027 [n3,replicate,s3,r127/1:/Table/74/1{-/3/4}] change replicas of r127 failed: storage/replica_raft.go:305: [n3,s3,r127/1:/Table/74/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:10.638440 92135 storage/queue.go:1027 [n3,replicate,s3,r39/4:/Table/6{0/2-2}] no removable replicas from range that needs a removal: [2:0, 4*:50] I190903 20:13:10.688995 92299 storage/replica_command.go:1430 [n3,replicate,s3,r122/3:/Table/74/1/5/{7-8}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r122:/Table/74/1/5/{7-8} [(n2,s2):2, (n3,s3):3, next=4, gen=88] I190903 20:13:10.699648 92299 storage/replica_raft.go:291 [n3,s3,r122/3:/Table/74/1/5/{7-8}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 20:13:10.716884 298 server/status/runtime.go:498 [n1] runtime stats: 240 MiB RSS, 971 goroutines, 47 MiB/45 MiB/138 MiB GO alloc/idle/total, 54 MiB/106 MiB CGO alloc/total, 13785.2 CGO/sec, 103.6/14.3 %(u/s)time, 0.7 %gc (19x), 96 MiB/96 MiB (r/w)net I190903 20:13:10.832843 88520 storage/replica_raft.go:291 [n1,s1,r118/2:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:10.832886 88520 storage/replica_raft.go:304 [n1,s1,r118/2:/Table/74/1/3/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.834150 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:10.834188 88520 storage/replica_command.go:979 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] could not promote [] to voter, rolling back: change replicas of r118 failed: storage/replica_raft.go:305: [n1,s1,r118/2:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:10.834220 88520 storage/queue.go:1027 [n1,replicate,s1,r118/2:/Table/74/1/3/{4-5}] change replicas of r118 failed: storage/replica_raft.go:305: [n1,s1,r118/2:/Table/74/1/3/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.835433 92711 storage/replica_command.go:1430 [n1,replicate,s1,r125/3:/Table/74/{1/6-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r125:/Table/74/{1/6-2} [(n2,s2):2, (n1,s1):3, next=4, gen=96] I190903 20:13:10.835965 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:10.840905 92711 storage/replica_raft.go:291 [n1,s1,r125/3:/Table/74/{1/6-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:10.847521 92717 storage/replica_command.go:1430 [n1,replicate,s1,r127/2:/Table/74/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r127:/Table/74/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=90] I190903 20:13:10.847603 89685 storage/store.go:2593 [n2,replicaGC,s2,r120/2:/Table/74/1{-/3/4}] removing replica r120/2 I190903 20:13:10.854312 89685 storage/replica_destroy.go:152 [n2,replicaGC,s2,r120/2:/Table/74/1{-/3/4}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.855263 92476 storage/store.go:2593 [n2,replicaGC,s2,r122/2:/Table/74/1/5/{7-8}] removing replica r122/2 I190903 20:13:10.855647 92476 storage/replica_destroy.go:152 [n2,replicaGC,s2,r122/2:/Table/74/1/5/{7-8}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.858078 92717 storage/replica_raft.go:291 [n1,s1,r127/2:/Table/74/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:10.860033 92775 storage/replica_command.go:1430 [n1,replicate,s1,r123/2:/Table/74/1/5/{8-9}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r123:/Table/74/1/5/{8-9} [(n3,s3):1, (n1,s1):2, next=3, gen=86] I190903 20:13:10.869321 92804 storage/store.go:2593 [n3,replicaGC,s3,r127/1:/Table/74/1{-/3/4}] removing replica r127/1 I190903 20:13:10.869716 92804 storage/replica_destroy.go:152 [n3,replicaGC,s3,r127/1:/Table/74/1{-/3/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.869756 92775 storage/replica_raft.go:291 [n1,s1,r123/2:/Table/74/1/5/{8-9}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:10.871386 92848 storage/replica_command.go:1430 [n1,replicate,s1,r119/3:/Table/74/1/{3/5-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r119:/Table/74/1/{3/5-5} [(n2,s2):2, (n1,s1):3, next=4, gen=86] I190903 20:13:10.881739 92848 storage/replica_raft.go:291 [n1,s1,r119/3:/Table/74/1/{3/5-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:10.881870 92859 storage/store.go:2593 [n3,replicaGC,s3,r123/1:/Table/74/1/5/{8-9}] removing replica r123/1 I190903 20:13:10.882211 92859 storage/replica_destroy.go:152 [n3,replicaGC,s3,r123/1:/Table/74/1/5/{8-9}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.887540 91729 storage/replica_command.go:1430 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r121:/Table/74/1/5{-/7} [(n1,s1):2, next=3, gen=85] E190903 20:13:10.888741 92949 storage/queue.go:1027 [n2,replicate,s2,r118/3:/Table/74/1/3/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 20:13:10.892016 91729 storage/replica_raft.go:291 [n1,s1,r121/2:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:10.893300 91729 storage/store_snapshot.go:995 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] sending LEARNER snapshot 0b742f16 at applied index 34 I190903 20:13:10.893577 91729 storage/store_snapshot.go:1038 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.894827 92954 storage/replica_raftstorage.go:793 [n2,s2,r121/3:{-}] applying LEARNER snapshot [id=0b742f16 index=34] I190903 20:13:10.899346 92954 storage/replica_raftstorage.go:814 [n2,s2,r121/3:/Table/74/1/5{-/7}] applied LEARNER snapshot [total=3ms ingestion=4@1ms id=0b742f16 index=34] I190903 20:13:10.901715 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:10.902281 91729 storage/replica_command.go:1430 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r121:/Table/74/1/5{-/7} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=86] I190903 20:13:10.905762 91729 storage/replica_raft.go:291 [n1,s1,r121/2:/Table/74/1/5{-/7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:10.906295 91729 storage/replica_command.go:1430 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r121:/Table/74/1/5{-/7} [(n1,s1):2, (n2,s2):3, next=4, gen=87] I190903 20:13:10.906346 92967 storage/store.go:2593 [n2,replicaGC,s2,r125/2:/Table/74/{1/6-2}] removing replica r125/2 I190903 20:13:10.906970 92967 storage/replica_destroy.go:152 [n2,replicaGC,s2,r125/2:/Table/74/{1/6-2}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.907099 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:10.911792 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:10.917887 91729 storage/replica_raft.go:291 [n1,s1,r121/2:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:10.917958 91729 storage/replica_raft.go:304 [n1,s1,r121/2:/Table/74/1/5{-/7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.919068 91729 storage/replica_command.go:979 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] could not promote [] to voter, rolling back: change replicas of r121 failed: storage/replica_raft.go:305: [n1,s1,r121/2:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:10.919109 91729 storage/queue.go:1027 [n1,replicate,s1,r121/2:/Table/74/1/5{-/7}] change replicas of r121 failed: storage/replica_raft.go:305: [n1,s1,r121/2:/Table/74/1/5{-/7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.919519 91649 storage/replica_command.go:1430 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r124:/Table/74/1/{5/9-6} [(n1,s1):2, next=3, gen=88] I190903 20:13:10.923769 91649 storage/replica_raft.go:291 [n1,s1,r124/2:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:10.925342 91649 storage/store_snapshot.go:995 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] sending LEARNER snapshot 0d80bae5 at applied index 35 I190903 20:13:10.925607 91649 storage/store_snapshot.go:1038 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:10.929407 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:10.930110 93081 storage/replica_raftstorage.go:793 [n2,s2,r124/3:{-}] applying LEARNER snapshot [id=0d80bae5 index=35] I190903 20:13:10.931254 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:10.933229 93081 storage/replica_raftstorage.go:814 [n2,s2,r124/3:/Table/74/1/{5/9-6}] applied LEARNER snapshot [total=3ms ingestion=4@3ms id=0d80bae5 index=35] I190903 20:13:10.933793 91649 storage/replica_command.go:1430 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] change replicas (add [(n2,s2):3] remove []): existing descriptor r124:/Table/74/1/{5/9-6} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=89] I190903 20:13:10.935084 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:10.937838 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:10.937905 91649 storage/replica_raft.go:291 [n1,s1,r124/2:/Table/74/1/{5/9-6}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:10.938506 91649 storage/replica_command.go:1430 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r124:/Table/74/1/{5/9-6} [(n1,s1):2, (n2,s2):3, next=4, gen=90] I190903 20:13:10.942601 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.945297 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.947708 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:10.950616 91649 storage/replica_raft.go:291 [n1,s1,r124/2:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:10.950662 91649 storage/replica_raft.go:304 [n1,s1,r124/2:/Table/74/1/{5/9-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.951402 91649 storage/replica_command.go:979 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] could not promote [] to voter, rolling back: change replicas of r124 failed: storage/replica_raft.go:305: [n1,s1,r124/2:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:10.951441 91649 storage/queue.go:1027 [n1,replicate,s1,r124/2:/Table/74/1/{5/9-6}] change replicas of r124 failed: storage/replica_raft.go:305: [n1,s1,r124/2:/Table/74/1/{5/9-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:10.952989 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.956139 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.956615 93118 storage/store.go:2593 [n2,replicaGC,s2,r119/2:/Table/74/1/{3/5-5}] removing replica r119/2 I190903 20:13:10.964498 93118 storage/replica_destroy.go:152 [n2,replicaGC,s2,r119/2:/Table/74/1/{3/5-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.958562 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:10.964777 93178 storage/replica_command.go:1430 [n2,replicate,s2,r118/3:/Table/74/1/3/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r118:/Table/74/1/3/{4-5} [(n1,s1):2, (n2,s2):3, next=4, gen=85] I190903 20:13:10.972632 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:10.976961 93178 storage/replica_raft.go:291 [n2,s2,r118/3:/Table/74/1/3/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:10.978620 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.980519 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:10.985534 93133 storage/store.go:2593 [n1,replicaGC,s1,r118/2:/Table/74/1/3/{4-5}] removing replica r118/2 I190903 20:13:10.985965 93133 storage/replica_destroy.go:152 [n1,replicaGC,s1,r118/2:/Table/74/1/3/{4-5}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:10.987653 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.005340 93325 storage/replica_command.go:1430 [n3,replicate,s3,r39/4:/Table/6{0/2-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r39:/Table/6{0/2-2} [(n2,s2):2, (n3,s3):4, next=5, gen=29] I190903 20:13:11.006899 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.008898 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.011741 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.013338 93325 storage/replica_raft.go:291 [n3,s3,r39/4:/Table/6{0/2-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):4] next=5 I190903 20:13:11.017871 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.020802 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.024765 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:11.029612 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.034497 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.042867 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:11.048654 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.052092 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.056011 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.064246 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.068638 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.075444 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:11.081418 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:11.084093 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.088370 93391 storage/store.go:2593 [n2,replicaGC,s2,r39/2:/Table/6{0/2-2}] removing replica r39/2 I190903 20:13:11.088887 93391 storage/replica_destroy.go:152 [n2,replicaGC,s2,r39/2:/Table/6{0/2-2}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:11.089139 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.091485 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.097455 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:11.099378 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.111844 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.113790 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 E190903 20:13:11.126779 93702 storage/queue.go:1027 [n2,replicate,s2,r124/3:/Table/74/1/{5/9-6}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 20:13:11.133015 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:11.134871 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) I190903 20:13:11.137986 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.141564 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.144421 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.164500 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.168942 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:11.206827 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:11.329999 1776 server/status/runtime.go:498 [n2] runtime stats: 241 MiB RSS, 961 goroutines, 42 MiB/51 MiB/138 MiB GO alloc/idle/total, 55 MiB/106 MiB CGO alloc/total, 13191.8 CGO/sec, 100.7/13.8 %(u/s)time, 0.7 %gc (18x), 97 MiB/97 MiB (r/w)net I190903 20:13:11.334056 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:11.337204 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:11.339002 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:11.459482 2354 server/status/runtime.go:498 [n3] runtime stats: 243 MiB RSS, 961 goroutines, 52 MiB/44 MiB/138 MiB GO alloc/idle/total, 56 MiB/109 MiB CGO alloc/total, 13184.4 CGO/sec, 100.8/13.8 %(u/s)time, 0.7 %gc (18x), 97 MiB/97 MiB (r/w)net I190903 20:13:11.511135 94689 storage/replica_command.go:1430 [n2,replicate,s2,r124/3:/Table/74/1/{5/9-6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r124:/Table/74/1/{5/9-6} [(n1,s1):2, (n2,s2):3, next=4, gen=90] I190903 20:13:11.520160 94689 storage/replica_raft.go:291 [n2,s2,r124/3:/Table/74/1/{5/9-6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:11.528240 94789 storage/store.go:2593 [n1,replicaGC,s1,r124/2:/Table/74/1/{5/9-6}] removing replica r124/2 I190903 20:13:11.528643 94789 storage/replica_destroy.go:152 [n1,replicaGC,s1,r124/2:/Table/74/1/{5/9-6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:11.533977 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 E190903 20:13:11.658148 95113 storage/queue.go:1027 [n2,replicate,s2,r121/3:/Table/74/1/5{-/7}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 20:13:11.804397 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:11.807926 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:11.809821 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:12.153049 96186 storage/replica_command.go:1430 [n2,replicate,s2,r121/3:/Table/74/1/5{-/7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r121:/Table/74/1/5{-/7} [(n1,s1):2, (n2,s2):3, next=4, gen=87] I190903 20:13:12.161104 96186 storage/replica_raft.go:291 [n2,s2,r121/3:/Table/74/1/5{-/7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:12.171096 96502 storage/store.go:2593 [n1,replicaGC,s1,r121/2:/Table/74/1/5{-/7}] removing replica r121/2 I190903 20:13:12.171483 96502 storage/replica_destroy.go:152 [n1,replicaGC,s1,r121/2:/Table/74/1/5{-/7}] removed 4 (0+4) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:12.405686 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 3 AND b = 4 I190903 20:13:12.409293 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) > (3, 4) AND a < 5 I190903 20:13:12.411157 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b < 7 I190903 20:13:12.415433 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 7 I190903 20:13:12.419050 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b = 8 I190903 20:13:12.422925 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a = 5 AND b > 8 I190903 20:13:12.427555 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:12.432996 78821 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col list partitioning - DEFAULT" WHERE (a, b) < (3, 4) === RUN TestRepartitioning/single_col_range_partitioning/single_col_range_partitioning_-_MAXVALUE I190903 20:13:12.455607 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=74 (multi col list partitioning - DEFAULT) version=5 mtime=2019-09-03 20:13:12.434859344 +0000 UTC I190903 20:13:12.467727 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 73, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col list partitioning - DEFAULT"]} I190903 20:13:12.482691 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col list partitioning - DEFAULT 74 4}] I190903 20:13:12.531660 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=74 (multi col list partitioning - DEFAULT) version=6 mtime=2019-09-03 20:13:12.531482432 +0000 UTC I190903 20:13:12.560778 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 75, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:12.568981 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 76, info: {TableName:data.public."single col range partitioning" Statement:CREATE TABLE "single col range partitioning" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root} I190903 20:13:12.570738 98683 storage/replica_command.go:284 [n2,split,s2,r57/1:/{Table/74/2-Max}] initiating a split of this range at key /Table/76 [r58] (zone config) I190903 20:13:12.576456 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:12.584056 98674 storage/replica_command.go:284 [n2,split,s2,r58/1:/{Table/76-Max}] initiating a split of this range at key /Table/76/1 [r59] (zone config) I190903 20:13:12.595627 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:12.601225 99334 storage/replica_command.go:284 [n2,split,s2,r59/1:/{Table/76/1-Max}] initiating a split of this range at key /Table/76/1/3 [r60] (zone config) I190903 20:13:12.610866 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:12.611307 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.622962 98759 storage/replica_command.go:284 [n2,split,s2,r60/1:/{Table/76/1/3-Max}] initiating a split of this range at key /Table/76/1/4 [r61] (zone config) I190903 20:13:12.639341 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.642771 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.648218 99745 storage/replica_command.go:1430 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r60:/Table/76/1/{3-4} [(n2,s2):1, next=2, gen=98] I190903 20:13:12.649194 99892 storage/replica_command.go:284 [n2,split,s2,r61/1:/{Table/76/1/4-Max}] initiating a split of this range at key /Table/76/2 [r62] (zone config) I190903 20:13:12.656380 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:12.663151 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.668402 99745 storage/replica_raft.go:291 [n2,s2,r60/1:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:12.668647 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.669402 99745 storage/store_snapshot.go:995 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] sending LEARNER snapshot c32622be at applied index 17 I190903 20:13:12.669589 99745 storage/store_snapshot.go:1038 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:12.671193 100038 storage/replica_raftstorage.go:793 [n3,s3,r60/2:{-}] applying LEARNER snapshot [id=c32622be index=17] I190903 20:13:12.672612 100038 storage/replica_raftstorage.go:814 [n3,s3,r60/2:/Table/76/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=c32622be index=17] I190903 20:13:12.672642 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:12.673930 99745 storage/replica_command.go:1430 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r60:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=99] I190903 20:13:12.677551 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.682153 99745 storage/replica_raft.go:291 [n2,s2,r60/1:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:12.682727 99745 storage/replica_command.go:1430 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r60:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=100] I190903 20:13:12.683529 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.687274 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.692276 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.696587 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.699796 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:12.703104 99745 storage/replica_raft.go:291 [n2,s2,r60/1:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:12.703147 99745 storage/replica_raft.go:304 [n2,s2,r60/1:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:12.704038 99745 storage/replica_command.go:979 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n2,s2,r60/1:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:12.704076 99745 storage/queue.go:1027 [n2,replicate,s2,r60/1:/Table/76/1/{3-4}] change replicas of r60 failed: storage/replica_raft.go:305: [n2,s2,r60/1:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:12.704462 98766 storage/replica_command.go:1430 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r61:/Table/76/{1/4-2} [(n2,s2):1, next=2, gen=99] I190903 20:13:12.704649 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:12.711024 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.729945 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.730547 98766 storage/replica_raft.go:291 [n2,s2,r61/1:/Table/76/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:12.732362 98766 storage/store_snapshot.go:995 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] sending LEARNER snapshot 8d1d3a31 at applied index 18 I190903 20:13:12.732903 98766 storage/store_snapshot.go:1038 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:12.734306 100091 storage/replica_raftstorage.go:793 [n1,s1,r61/2:{-}] applying LEARNER snapshot [id=8d1d3a31 index=18] I190903 20:13:12.739322 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.745508 100091 storage/replica_raftstorage.go:814 [n1,s1,r61/2:/Table/76/{1/4-2}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=8d1d3a31 index=18] I190903 20:13:12.746165 98766 storage/replica_command.go:1430 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r61:/Table/76/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=100] I190903 20:13:12.746516 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.749356 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:12.755537 98766 storage/replica_raft.go:291 [n2,s2,r61/1:/Table/76/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:12.756394 98766 storage/replica_command.go:1430 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r61:/Table/76/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=101] I190903 20:13:12.757556 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.761126 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.766580 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.772571 98766 storage/replica_raft.go:291 [n2,s2,r61/1:/Table/76/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:12.772614 98766 storage/replica_raft.go:304 [n2,s2,r61/1:/Table/76/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:12.773414 98766 storage/replica_command.go:979 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r61 failed: storage/replica_raft.go:305: [n2,s2,r61/1:/Table/76/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:12.773448 98766 storage/queue.go:1027 [n2,replicate,s2,r61/1:/Table/76/{1/4-2}] change replicas of r61 failed: storage/replica_raft.go:305: [n2,s2,r61/1:/Table/76/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:12.828178 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.831598 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.835094 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.838676 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.842428 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.850164 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.855147 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.859537 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.863119 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.872163 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.875331 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.879702 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.882830 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.888243 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.891463 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.899034 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.901990 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:12.915643 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.918498 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 20:13:12.935427 100846 storage/queue.go:1027 [n1,replicate,s1,r61/2:/Table/76/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:12.940160 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:12.949413 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 20:13:12.955678 101284 storage/queue.go:1027 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:12.989621 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:13.114051 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:13.117748 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:13.121838 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:13.127392 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 20:13:13.125905104 +0000 UTC I190903 20:13:13.136288 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:13.138287 101794 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=2:1567541894.717203516,0, refcount=0 I190903 20:13:13.138788 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 1}] I190903 20:13:13.193632 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=76 (single col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 20:13:13.193419256 +0000 UTC I190903 20:13:13.199693 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 2}] I190903 20:13:13.202824 102215 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=3:1567541898.663224241,0, refcount=0 I190903 20:13:13.205555 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:13.224508 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 20:13:13.222580207 +0000 UTC I190903 20:13:13.229175 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 76, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:ALTER TABLE "single col range partitioning - MAXVALUE" PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:13.232053 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 3}] I190903 20:13:13.232697 102000 sql/lease.go:923 new lease: 76("single col range partitioning - MAXVALUE") ver=4:1567541898.664224241,0, refcount=0 I190903 20:13:13.251890 102575 storage/replica_command.go:1430 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r60:/Table/76/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=100] I190903 20:13:13.264862 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:13.266706 102575 storage/replica_raft.go:291 [n3,s3,r60/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:13.270919 103136 storage/queue.go:1027 [n2,merge,s2,r59/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 20:13:13.280193 103147 storage/store.go:2593 [n2,replicaGC,s2,r60/1:/Table/76/1/{3-4}] removing replica r60/1 I190903 20:13:13.280590 103147 storage/replica_destroy.go:152 [n2,replicaGC,s2,r60/1:/Table/76/1/{3-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:13.280858 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:13.282144 103334 storage/replica_command.go:284 [n1,split,s1,r61/2:/Table/76/{1/4-2}] initiating a split of this range at key /Table/76/1/5 [r80] (zone config) E190903 20:13:13.334138 103593 storage/queue.go:1027 [n2,merge,s2,r59/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 20:13:13.339767 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 76, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:13.339873 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 20:13:13.345135 103487 storage/queue.go:1027 [n2,merge,s2,r59/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 E190903 20:13:13.356404 103978 storage/queue.go:1027 [n2,replicate,s2,r61/1:/Table/76/1/{4-5}] no removable replicas from range that needs a removal: [1*:33, 2:0] I190903 20:13:13.367216 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.372880 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.375805 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.379965 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.381606 104104 storage/replica_command.go:1430 [n2,replicate,s2,r40/2:/Table/64{-/1}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r40:/Table/64{-/1} [(n2,s2):2, next=3, gen=36] I190903 20:13:13.385669 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.388739 104104 storage/replica_raft.go:291 [n2,s2,r40/2:/Table/64{-/1}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:13:13.389997 104104 storage/store_snapshot.go:995 [n2,replicate,s2,r40/2:/Table/64{-/1}] sending LEARNER snapshot c10c0300 at applied index 38 I190903 20:13:13.390264 104104 storage/store_snapshot.go:1038 [n2,replicate,s2,r40/2:/Table/64{-/1}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:13.390621 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.391334 104202 storage/replica_raftstorage.go:793 [n3,s3,r40/3:{-}] applying LEARNER snapshot [id=c10c0300 index=38] I190903 20:13:13.392583 104202 storage/replica_raftstorage.go:814 [n3,s3,r40/3:/Table/64{-/1}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=c10c0300 index=38] I190903 20:13:13.393633 104104 storage/replica_command.go:1430 [n2,replicate,s2,r40/2:/Table/64{-/1}] change replicas (add [(n3,s3):3] remove []): existing descriptor r40:/Table/64{-/1} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=37] I190903 20:13:13.398037 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.402476 104104 storage/replica_raft.go:291 [n2,s2,r40/2:/Table/64{-/1}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 20:13:13.402626 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.406161 104104 storage/replica_command.go:1430 [n2,replicate,s2,r40/2:/Table/64{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r40:/Table/64{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=38] I190903 20:13:13.408369 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.413012 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.418540 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.427416 104104 storage/replica_raft.go:291 [n2,s2,r40/2:/Table/64{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:13:13.427465 104104 storage/replica_raft.go:304 [n2,s2,r40/2:/Table/64{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:13.427936 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.429071 104104 storage/replica_command.go:979 [n2,replicate,s2,r40/2:/Table/64{-/1}] could not promote [] to voter, rolling back: change replicas of r40 failed: storage/replica_raft.go:305: [n2,s2,r40/2:/Table/64{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:13.430269 104104 storage/queue.go:1027 [n2,replicate,s2,r40/2:/Table/64{-/1}] change replicas of r40 failed: storage/replica_raft.go:305: [n2,s2,r40/2:/Table/64{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:13.432214 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.439464 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.444078 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.449653 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.460388 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:13.466234 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.468558 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.471978 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.474209 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.478378 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.483347 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.498347 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.512905 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:13.518193 104906 storage/replica_command.go:1430 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r60:/Table/76/1/{3-4} [(n3,s3):2, next=3, gen=101] E190903 20:13:13.525545 104779 storage/queue.go:1027 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:11] I190903 20:13:13.529587 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.535428 104906 storage/replica_raft.go:291 [n3,s3,r60/2:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:13.537293 104906 storage/store_snapshot.go:995 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] sending LEARNER snapshot b625e255 at applied index 30 I190903 20:13:13.537514 104906 storage/store_snapshot.go:1038 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:13.539056 104792 storage/replica_raftstorage.go:793 [n1,s1,r60/3:{-}] applying LEARNER snapshot [id=b625e255 index=30] I190903 20:13:13.540444 104792 storage/replica_raftstorage.go:814 [n1,s1,r60/3:/Table/76/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=b625e255 index=30] I190903 20:13:13.541384 104906 storage/replica_command.go:1430 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] change replicas (add [(n1,s1):3] remove []): existing descriptor r60:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=102] I190903 20:13:13.548373 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.569583 104906 storage/replica_raft.go:291 [n3,s3,r60/2:/Table/76/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:13.570148 104906 storage/replica_command.go:1430 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r60:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=103] I190903 20:13:13.585195 104906 storage/replica_raft.go:291 [n3,s3,r60/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:13.585641 104906 storage/replica_raft.go:304 [n3,s3,r60/2:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:13.586663 104906 storage/replica_command.go:979 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:13.586703 104906 storage/queue.go:1027 [n3,replicate,s3,r60/2:/Table/76/1/{3-4}] change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:13.588675 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:13.591600 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:13.594969 105191 storage/replica_command.go:1430 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r59:/Table/76/1{-/3} [(n2,s2):1, next=2, gen=97] W190903 20:13:13.595327 105030 storage/replica_command.go:1917 [n3,s3,r60/2:/Table/76/1/{3-4}] while transferring lease: unable to find store 2 in range r60:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=103] I190903 20:13:13.596352 105030 storage/replica_command.go:1430 [n3,s3,r60/2:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r60:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=103] I190903 20:13:13.603441 105030 storage/replica_raft.go:291 [n3,s3,r60/2:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:13.603481 105030 storage/replica_raft.go:304 [n3,s3,r60/2:/Table/76/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:13.604264 105191 storage/replica_raft.go:291 [n2,s2,r59/1:/Table/76/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:13.604343 105030 storage/replica_command.go:979 [n3,s3,r60/2:/Table/76/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) W190903 20:13:13.604411 105030 storage/replica_command.go:2033 [n3,s3,r60/2:/Table/76/1/{3-4}] while removing target n3,s3: change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:13.604727 105190 storage/queue.go:1027 [n2,merge,s2,r59/1:/Table/76/1{-/3}] change replicas of r60 failed: storage/replica_raft.go:305: [n3,s3,r60/2:/Table/76/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:13.605266 105191 storage/store_snapshot.go:995 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] sending LEARNER snapshot 83e1c39d at applied index 21 I190903 20:13:13.605883 105191 storage/store_snapshot.go:1038 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:13.606714 105230 storage/replica_raftstorage.go:793 [n1,s1,r59/2:{-}] applying LEARNER snapshot [id=83e1c39d index=21] I190903 20:13:13.608061 105230 storage/replica_raftstorage.go:814 [n1,s1,r59/2:/Table/76/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=83e1c39d index=21] I190903 20:13:13.608649 105191 storage/replica_command.go:1430 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r59:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=98] I190903 20:13:13.617579 105191 storage/replica_raft.go:291 [n2,s2,r59/1:/Table/76/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:13.618329 105191 storage/replica_command.go:1430 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r59:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=99] I190903 20:13:13.632223 105191 storage/replica_raft.go:291 [n2,s2,r59/1:/Table/76/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:13.632266 105191 storage/replica_raft.go:304 [n2,s2,r59/1:/Table/76/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:13.633319 105191 storage/replica_command.go:979 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] could not promote [] to voter, rolling back: change replicas of r59 failed: storage/replica_raft.go:305: [n2,s2,r59/1:/Table/76/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:13.633363 105191 storage/queue.go:1027 [n2,replicate,s2,r59/1:/Table/76/1{-/3}] change replicas of r59 failed: storage/replica_raft.go:305: [n2,s2,r59/1:/Table/76/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:13.660724 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:13.727595 105515 storage/replica_command.go:1430 [n2,replicate,s2,r61/1:/Table/76/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r61:/Table/76/1/{4-5} [(n2,s2):1, (n1,s1):2, next=3, gen=102] I190903 20:13:13.739313 105515 storage/replica_raft.go:291 [n2,s2,r61/1:/Table/76/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):1] next=3 I190903 20:13:13.750038 105655 storage/store.go:2593 [n1,replicaGC,s1,r61/2:/Table/76/1/{4-5}] removing replica r61/2 I190903 20:13:13.750422 105655 storage/replica_destroy.go:152 [n1,replicaGC,s1,r61/2:/Table/76/1/{4-5}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:13.800715 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 E190903 20:13:14.020371 106220 storage/queue.go:1027 [n1,replicate,s1,r60/3:/Table/76/1/{3-4}] no removable replicas from range that needs a removal: [2:0, 3*:41] E190903 20:13:14.046116 106384 storage/queue.go:1027 [n1,replicate,s1,r59/2:/Table/76/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:29] I190903 20:13:14.071087 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 20:13:14.087643 106382 storage/queue.go:1027 [n2,merge,s2,r59/1:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2,(n1,s1):3 E190903 20:13:14.100704 106660 storage/queue.go:1027 [n2,replicate,s2,r80/1:/Table/76/{1/5-2}] no removable replicas from range that needs a removal: [1*:14, 2:0] I190903 20:13:14.127055 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:14.134572 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:14.181728 2349 storage/compactor/compactor.go:325 [n3,s3,compactor] purging suggested compaction for range /System/tsd - /System/"tse" that contains live data E190903 20:13:14.236927 107037 storage/queue.go:1027 [n3,replicate,s3,r40/3:/Table/64{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:45] E190903 20:13:14.275310 107133 storage/queue.go:1027 [n1,merge,s1,r59/2:/Table/76/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2,(n1,s1):3 I190903 20:13:14.275993 107134 storage/replica_command.go:1430 [n1,replicate,s1,r59/2:/Table/76/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r59:/Table/76/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=99] I190903 20:13:14.280377 107134 storage/replica_raft.go:291 [n1,s1,r59/2:/Table/76/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:14.291330 106626 storage/store.go:2593 [n2,replicaGC,s2,r59/1:/Table/76/1{-/3}] removing replica r59/1 I190903 20:13:14.291705 106626 storage/replica_destroy.go:152 [n2,replicaGC,s2,r59/1:/Table/76/1{-/3}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:14.483424 107632 storage/replica_command.go:1430 [n1,replicate,s1,r60/3:/Table/76/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r60:/Table/76/1/{3-4} [(n3,s3):2, (n1,s1):3, next=4, gen=103] I190903 20:13:14.487325 107632 storage/replica_raft.go:291 [n1,s1,r60/3:/Table/76/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:14.496121 107171 storage/store.go:2593 [n3,replicaGC,s3,r60/2:/Table/76/1/{3-4}] removing replica r60/2 I190903 20:13:14.496550 107171 storage/replica_destroy.go:152 [n3,replicaGC,s3,r60/2:/Table/76/1/{3-4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:14.609929 108000 storage/replica_command.go:1430 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] change replicas (add [(n3,s3):4LEARNER] remove []): existing descriptor r55:/Table/6{2/2-4} [(n2,s2):3, next=4, gen=35] I190903 20:13:14.617284 108000 storage/replica_raft.go:291 [n2,s2,r55/3:/Table/6{2/2-4}] proposing ADD_REPLICA[(n3,s3):4LEARNER]: after=[(n2,s2):3 (n3,s3):4LEARNER] next=5 I190903 20:13:14.621897 108000 storage/store_snapshot.go:995 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] sending LEARNER snapshot 8402674d at applied index 53 I190903 20:13:14.622068 108000 storage/store_snapshot.go:1038 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] streamed snapshot to (n3,s3):4LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:14.623156 107690 storage/replica_raftstorage.go:793 [n3,s3,r55/4:{-}] applying LEARNER snapshot [id=8402674d index=53] I190903 20:13:14.624467 107690 storage/replica_raftstorage.go:814 [n3,s3,r55/4:/Table/6{2/2-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=8402674d index=53] I190903 20:13:14.625519 108000 storage/replica_command.go:1430 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] change replicas (add [(n3,s3):4] remove []): existing descriptor r55:/Table/6{2/2-4} [(n2,s2):3, (n3,s3):4LEARNER, next=5, gen=36] I190903 20:13:14.631922 108000 storage/replica_raft.go:291 [n2,s2,r55/3:/Table/6{2/2-4}] proposing ADD_REPLICA[(n3,s3):4]: after=[(n2,s2):3 (n3,s3):4] next=5 I190903 20:13:14.633208 108000 storage/replica_command.go:1430 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r55:/Table/6{2/2-4} [(n2,s2):3, (n3,s3):4, next=5, gen=37] I190903 20:13:14.651209 108000 storage/replica_raft.go:291 [n2,s2,r55/3:/Table/6{2/2-4}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 E190903 20:13:14.651252 108000 storage/replica_raft.go:304 [n2,s2,r55/3:/Table/6{2/2-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:13:14.652309 108000 storage/replica_command.go:979 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] could not promote [] to voter, rolling back: change replicas of r55 failed: storage/replica_raft.go:305: [n2,s2,r55/3:/Table/6{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) E190903 20:13:14.652341 108000 storage/queue.go:1027 [n2,replicate,s2,r55/3:/Table/6{2/2-4}] change replicas of r55 failed: storage/replica_raft.go:305: [n2,s2,r55/3:/Table/6{2/2-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 to remove self (leaseholder) I190903 20:13:14.678089 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:14.687888 108118 storage/replica_command.go:1430 [n3,replicate,s3,r40/3:/Table/64{-/1}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r40:/Table/64{-/1} [(n2,s2):2, (n3,s3):3, next=4, gen=38] I190903 20:13:14.696915 108118 storage/replica_raft.go:291 [n3,s3,r40/3:/Table/64{-/1}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:13:14.741112 108018 storage/queue.go:1027 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:17] I190903 20:13:14.761340 108286 storage/store.go:2593 [n2,replicaGC,s2,r40/2:/Table/64{-/1}] removing replica r40/2 I190903 20:13:14.761878 108286 storage/replica_destroy.go:152 [n2,replicaGC,s2,r40/2:/Table/64{-/1}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:15.128576 109450 storage/replica_command.go:609 [n1,merge,s1,r59/2:/Table/76/1{-/3}] initiating a merge of r60:/Table/76/1/{3-4} [(n1,s1):3, next=4, gen=104] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:15.134920 191 storage/store.go:2593 [n1,s1,r59/2:/Table/76/1{-/3}] removing replica r60/3 I190903 20:13:15.213235 109688 storage/replica_command.go:1430 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r80:/Table/76/{1/5-2} [(n2,s2):1, (n1,s1):2, next=3, gen=102] I190903 20:13:15.216990 109688 storage/replica_raft.go:291 [n1,s1,r80/2:/Table/76/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:15.217914 109688 storage/replica_command.go:1430 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r80:/Table/76/{1/5-2} [(n1,s1):2, next=3, gen=103] I190903 20:13:15.227546 109688 storage/replica_raft.go:291 [n1,s1,r80/2:/Table/76/{1/5-2}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:15.228575 109688 storage/store_snapshot.go:995 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] sending LEARNER snapshot 652c45a1 at applied index 22 I190903 20:13:15.228746 109688 storage/store_snapshot.go:1038 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:15.229901 109520 storage/replica_raftstorage.go:793 [n3,s3,r80/3:{-}] applying LEARNER snapshot [id=652c45a1 index=22] I190903 20:13:15.231202 109520 storage/replica_raftstorage.go:814 [n3,s3,r80/3:/Table/76/{1/5-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=652c45a1 index=22] I190903 20:13:15.231629 109688 storage/replica_command.go:1430 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] change replicas (add [(n3,s3):3] remove []): existing descriptor r80:/Table/76/{1/5-2} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=104] I190903 20:13:15.234331 109725 storage/store.go:2593 [n2,replicaGC,s2,r80/1:/Table/76/{1/5-2}] removing replica r80/1 I190903 20:13:15.234641 109725 storage/replica_destroy.go:152 [n2,replicaGC,s2,r80/1:/Table/76/{1/5-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:15.235249 109688 storage/replica_raft.go:291 [n1,s1,r80/2:/Table/76/{1/5-2}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 20:13:15.235775 109688 storage/replica_command.go:1430 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r80:/Table/76/{1/5-2} [(n1,s1):2, (n3,s3):3, next=4, gen=105] I190903 20:13:15.245291 109688 storage/replica_raft.go:291 [n1,s1,r80/2:/Table/76/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 20:13:15.245332 109688 storage/replica_raft.go:304 [n1,s1,r80/2:/Table/76/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:15.246156 109688 storage/replica_command.go:979 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r80 failed: storage/replica_raft.go:305: [n1,s1,r80/2:/Table/76/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:15.246198 109688 storage/queue.go:1027 [n1,replicate,s1,r80/2:/Table/76/{1/5-2}] change replicas of r80 failed: storage/replica_raft.go:305: [n1,s1,r80/2:/Table/76/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:15.289982 108408 storage/replica_command.go:1430 [n3,replicate,s3,r55/4:/Table/6{2/2-4}] change replicas (add [] remove [(n2,s2):3]): existing descriptor r55:/Table/6{2/2-4} [(n2,s2):3, (n3,s3):4, next=5, gen=37] I190903 20:13:15.297076 108408 storage/replica_raft.go:291 [n3,s3,r55/4:/Table/6{2/2-4}] proposing REMOVE_REPLICA[(n2,s2):3]: after=[(n3,s3):4] next=5 I190903 20:13:15.344764 109983 storage/replica_command.go:1430 [n3,replicate,s3,r80/3:/Table/76/{1/5-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r80:/Table/76/{1/5-2} [(n1,s1):2, (n3,s3):3, next=4, gen=105] I190903 20:13:15.351770 109983 storage/replica_raft.go:291 [n3,s3,r80/3:/Table/76/{1/5-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 20:13:15.363002 109824 storage/store.go:2593 [n2,replicaGC,s2,r55/3:/Table/6{2/2-4}] removing replica r55/3 I190903 20:13:15.363434 109824 storage/replica_destroy.go:152 [n2,replicaGC,s2,r55/3:/Table/6{2/2-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:15.411001 110117 storage/store.go:2593 [n1,replicaGC,s1,r80/2:/Table/76/{1/5-2}] removing replica r80/2 I190903 20:13:15.411373 110117 storage/replica_destroy.go:152 [n1,replicaGC,s1,r80/2:/Table/76/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:15.684544 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:15.687934 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:15.689716 97255 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 === RUN TestRepartitioning/single_col_range_partitioning_-_MAXVALUE/single_col_range_partitioning I190903 20:13:15.704804 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=76 (single col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 20:13:15.693055228 +0000 UTC I190903 20:13:15.712694 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 75, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning - MAXVALUE"]} I190903 20:13:15.714739 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning - MAXVALUE 76 4}] I190903 20:13:15.715033 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:15.770436 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=76 (single col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 20:13:15.770276818 +0000 UTC I190903 20:13:15.823835 1727 storage/queue.go:518 [n2,s2,r3/3:/System/{NodeLive…-tsd}] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:15.825740 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 77, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:15.849370 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 78, info: {TableName:data.public."single col range partitioning - MAXVALUE" Statement:CREATE TABLE "single col range partitioning - MAXVALUE" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p4 VALUES FROM (minvalue) TO (4), PARTITION p5 VALUES FROM (4) TO (5), PARTITION px VALUES FROM (5) TO (maxvalue)) User:root} I190903 20:13:15.854603 112512 storage/replica_command.go:284 [n2,split,s2,r62/1:/{Table/76/2-Max}] initiating a split of this range at key /Table/78 [r63] (zone config) I190903 20:13:15.946659 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:15.953274 113004 storage/replica_command.go:284 [n2,split,s2,r63/1:/{Table/78-Max}] initiating a split of this range at key /Table/78/1 [r64] (zone config) I190903 20:13:15.976306 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p5 OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:15.996369 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION px OF INDEX data.public."single col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:15.997886 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:15.997993 113549 storage/replica_command.go:284 [n2,split,s2,r64/1:/{Table/78/1-Max}] initiating a split of this range at key /Table/78/1/4 [r65] (zone config) I190903 20:13:16.021010 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.029559 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.031891 113965 storage/replica_command.go:1430 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r64:/Table/78/1{-/4} [(n2,s2):1, next=2, gen=102] I190903 20:13:16.031959 113993 storage/replica_command.go:284 [n2,split,s2,r65/1:/{Table/78/1/4-Max}] initiating a split of this range at key /Table/78/1/5 [r66] (zone config) I190903 20:13:16.041614 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.043748 113965 storage/replica_raft.go:291 [n2,s2,r64/1:/Table/78/1{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:16.044866 113965 storage/store_snapshot.go:995 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] sending LEARNER snapshot 42345327 at applied index 17 I190903 20:13:16.045018 113965 storage/store_snapshot.go:1038 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:16.045203 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.046228 114073 storage/replica_command.go:284 [n2,split,s2,r66/1:/{Table/78/1/5-Max}] initiating a split of this range at key /Table/78/2 [r67] (zone config) I190903 20:13:16.049707 114054 storage/replica_raftstorage.go:793 [n1,s1,r64/2:{-}] applying LEARNER snapshot [id=42345327 index=17] I190903 20:13:16.051207 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.053397 114054 storage/replica_raftstorage.go:814 [n1,s1,r64/2:/Table/78/1{-/4}] applied LEARNER snapshot [total=4ms ingestion=4@3ms id=42345327 index=17] I190903 20:13:16.054675 113965 storage/replica_command.go:1430 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r64:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=103] I190903 20:13:16.058434 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.062913 113965 storage/replica_raft.go:291 [n2,s2,r64/1:/Table/78/1{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:16.064268 113965 storage/replica_command.go:1430 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r64:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=104] I190903 20:13:16.064501 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.085906 113965 storage/replica_raft.go:291 [n2,s2,r64/1:/Table/78/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:16.085953 113965 storage/replica_raft.go:304 [n2,s2,r64/1:/Table/78/1{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:16.087474 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.087591 113965 storage/replica_command.go:979 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] could not promote [] to voter, rolling back: change replicas of r64 failed: storage/replica_raft.go:305: [n2,s2,r64/1:/Table/78/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:16.087620 113965 storage/queue.go:1027 [n2,replicate,s2,r64/1:/Table/78/1{-/4}] change replicas of r64 failed: storage/replica_raft.go:305: [n2,s2,r64/1:/Table/78/1{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:16.088024 114078 storage/replica_command.go:1430 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r66:/Table/78/{1/5-2} [(n2,s2):1, next=2, gen=104] I190903 20:13:16.109439 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.117464 114078 storage/replica_raft.go:291 [n2,s2,r66/1:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:16.121438 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.151718 114078 storage/store_snapshot.go:995 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] sending LEARNER snapshot d5ed9518 at applied index 17 I190903 20:13:16.151874 114078 storage/store_snapshot.go:1038 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:16.155535 114336 storage/replica_raftstorage.go:793 [n3,s3,r66/2:{-}] applying LEARNER snapshot [id=d5ed9518 index=17] I190903 20:13:16.156973 114336 storage/replica_raftstorage.go:814 [n3,s3,r66/2:/Table/78/{1/5-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=d5ed9518 index=17] I190903 20:13:16.157640 114078 storage/replica_command.go:1430 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r66:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=105] I190903 20:13:16.164548 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.172788 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.178097 114078 storage/replica_raft.go:291 [n2,s2,r66/1:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:16.178625 114078 storage/replica_command.go:1430 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r66:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=106] I190903 20:13:16.180797 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.192392 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.214421 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.216397 114078 storage/replica_raft.go:291 [n2,s2,r66/1:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:16.216430 114078 storage/replica_raft.go:304 [n2,s2,r66/1:/Table/78/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:16.217226 114078 storage/replica_command.go:979 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r66 failed: storage/replica_raft.go:305: [n2,s2,r66/1:/Table/78/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:16.217272 114078 storage/queue.go:1027 [n2,replicate,s2,r66/1:/Table/78/{1/5-2}] change replicas of r66 failed: storage/replica_raft.go:305: [n2,s2,r66/1:/Table/78/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:16.220277 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.225168 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.228961 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.242037 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.255175 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.261118 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.265755 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.270925 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.281344 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.284321 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.293407 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.300407 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.310523 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.325882 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.351414 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 20:13:16.373980 115226 storage/queue.go:1027 [n3,replicate,s3,r66/2:/Table/78/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:16.392575 115297 storage/replica_command.go:1430 [n3,replicate,s3,r66/2:/Table/78/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r66:/Table/78/{1/5-2} [(n2,s2):1, (n3,s3):2, next=3, gen=106] I190903 20:13:16.398549 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.406845 115297 storage/replica_raft.go:291 [n3,s3,r66/2:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:16.423747 115224 storage/consistency_queue.go:133 [n2,consistencyChecker,s2,r66/1:/Table/78/{1/5-2}] could not get replica descriptor: r66 was not found on s2 E190903 20:13:16.423788 115224 storage/queue.go:1027 [n2,consistencyChecker,s2,r66/1:/Table/78/{1/5-2}] could not get replica descriptor: r66 was not found on s2 I190903 20:13:16.430048 114475 storage/store.go:2593 [n2,replicaGC,s2,r66/1:/Table/78/{1/5-2}] removing replica r66/1 I190903 20:13:16.430453 114475 storage/replica_destroy.go:152 [n2,replicaGC,s2,r66/1:/Table/78/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:16.471233 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 E190903 20:13:16.519492 115663 storage/queue.go:1027 [n1,replicate,s1,r64/2:/Table/78/1{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:13:16.594301 115601 storage/replica_command.go:1430 [n1,replicate,s1,r64/2:/Table/78/1{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r64:/Table/78/1{-/4} [(n2,s2):1, (n1,s1):2, next=3, gen=104] I190903 20:13:16.599676 115601 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/78/1{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:16.608790 115810 storage/store.go:2593 [n2,replicaGC,s2,r64/1:/Table/78/1{-/4}] removing replica r64/1 I190903 20:13:16.609180 115810 storage/replica_destroy.go:152 [n2,replicaGC,s2,r64/1:/Table/78/1{-/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:16.610728 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.616273 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.892143 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a < 4 I190903 20:13:16.894339 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a >= 4 AND a < 5 I190903 20:13:16.897819 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning - MAXVALUE" WHERE a > 5 I190903 20:13:16.902324 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=78 (single col range partitioning) version=2 mtime=2019-09-03 20:13:16.900920722 +0000 UTC I190903 20:13:16.910111 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 1}] I190903 20:13:16.911437 114624 sql/lease.go:923 new lease: 78("single col range partitioning") ver=2:1567541904.931032898,0, refcount=0 I190903 20:13:16.960135 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=78 (single col range partitioning) version=3 mtime=2019-09-03 20:13:16.959957884 +0000 UTC I190903 20:13:16.965172 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 2}] I190903 20:13:16.968290 116912 sql/lease.go:923 new lease: 78("single col range partitioning") ver=3:1567541904.932032898,0, refcount=0 I190903 20:13:16.992199 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=78 (single col range partitioning) version=4 mtime=2019-09-03 20:13:16.990018315 +0000 UTC I190903 20:13:16.999394 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 78, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:17.002656 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 3}] I190903 20:13:17.002914 117718 storage/replica_command.go:1430 [n2,s2,r67/1:/{Table/78/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r67:/{Table/78/2-Max} [(n2,s2):1, next=2, gen=104] I190903 20:13:17.010630 117866 storage/replica_command.go:1430 [n3,s3,r66/2:/Table/78/{1/5-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r66:/Table/78/{1/5-2} [(n3,s3):2, next=3, gen=107] I190903 20:13:17.015765 117563 sql/lease.go:923 new lease: 78("single col range partitioning") ver=4:1567541910.854813072,0, refcount=0 I190903 20:13:17.019469 117866 storage/replica_raft.go:291 [n3,s3,r66/2:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:13:17.020684 117866 storage/store_snapshot.go:995 [n3,s3,r66/2:/Table/78/{1/5-2}] sending LEARNER snapshot 5b082fcc at applied index 31 I190903 20:13:17.021396 117866 storage/store_snapshot.go:1038 [n3,s3,r66/2:/Table/78/{1/5-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:17.022534 117917 storage/replica_raftstorage.go:793 [n2,s2,r66/3:{-}] applying LEARNER snapshot [id=5b082fcc index=31] I190903 20:13:17.025995 117718 storage/replica_raft.go:291 [n2,s2,r67/1:/{Table/78/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:17.028026 117718 storage/store_snapshot.go:995 [n2,s2,r67/1:/{Table/78/2-Max}] sending LEARNER snapshot d17d9c38 at applied index 18 I190903 20:13:17.028193 117718 storage/store_snapshot.go:1038 [n2,s2,r67/1:/{Table/78/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:17.029122 117920 storage/replica_raftstorage.go:793 [n3,s3,r67/2:{-}] applying LEARNER snapshot [id=d17d9c38 index=18] I190903 20:13:17.032648 117917 storage/replica_raftstorage.go:814 [n2,s2,r66/3:/Table/78/{1/5-2}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=5b082fcc index=31] I190903 20:13:17.035645 117866 storage/replica_command.go:1430 [n3,s3,r66/2:/Table/78/{1/5-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r66:/Table/78/{1/5-2} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=108] I190903 20:13:17.043639 117920 storage/replica_raftstorage.go:814 [n3,s3,r67/2:/{Table/78/2-Max}] applied LEARNER snapshot [total=14ms ingestion=4@14ms id=d17d9c38 index=18] I190903 20:13:17.044942 117718 storage/replica_command.go:1430 [n2,s2,r67/1:/{Table/78/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r67:/{Table/78/2-Max} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=105] I190903 20:13:17.047465 117866 storage/replica_raft.go:291 [n3,s3,r66/2:/Table/78/{1/5-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 E190903 20:13:17.051770 118036 storage/queue.go:1027 [n2,replicate,s2,r66/3:/Table/78/{1/5-2}] no removable replicas from range that needs a removal: [2:0, 3*:35] I190903 20:13:17.056559 117718 storage/replica_raft.go:291 [n2,s2,r67/1:/{Table/78/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 E190903 20:13:17.066778 117179 storage/queue.go:1027 [n3,replicate,s3,r67/2:/{Table/78/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 20:13:17.067702 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:17.077326 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:17.080474 118199 storage/replica_command.go:284 [n1,split,s1,r64/2:/Table/78/1{-/4}] initiating a split of this range at key /Table/78/1/3 [r101] (zone config) I190903 20:13:17.095130 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 78, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:17.095260 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.103858 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.108603 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.116843 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.119071 118557 storage/replica_command.go:1430 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r64:/Table/78/1{-/3} [(n1,s1):2, next=3, gen=106] I190903 20:13:17.121615 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.133409 118557 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/78/1{-/3}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:17.137165 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.138083 118557 storage/store_snapshot.go:995 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] sending LEARNER snapshot c26a2c2c at applied index 36 I190903 20:13:17.138311 118557 storage/store_snapshot.go:1038 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:17.139660 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.140212 119128 storage/replica_raftstorage.go:793 [n2,s2,r64/3:{-}] applying LEARNER snapshot [id=c26a2c2c index=36] I190903 20:13:17.151126 119128 storage/replica_raftstorage.go:814 [n2,s2,r64/3:/Table/78/1{-/3}] applied LEARNER snapshot [total=10ms ingestion=4@9ms id=c26a2c2c index=36] I190903 20:13:17.154349 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.156392 118557 storage/replica_command.go:1430 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] change replicas (add [(n2,s2):3] remove []): existing descriptor r64:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=107] I190903 20:13:17.164666 118557 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/78/1{-/3}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:17.174063 118557 storage/replica_command.go:1430 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r64:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=108] I190903 20:13:17.180007 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.180259 119107 storage/replica_command.go:1430 [n2,s2,r66/3:/Table/78/{1/5-2}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r66:/Table/78/{1/5-2} [(n3,s3):2, (n2,s2):3, next=4, gen=109] I190903 20:13:17.185440 119083 storage/replica_command.go:1430 [n3,s3,r67/2:/{Table/78/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r67:/{Table/78/2-Max} [(n2,s2):1, (n3,s3):2, next=3, gen=106] I190903 20:13:17.194175 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.211578 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.213692 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.221418 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.224290 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.226739 118557 storage/replica_raft.go:291 [n1,s1,r64/2:/Table/78/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:17.226849 118557 storage/replica_raft.go:304 [n1,s1,r64/2:/Table/78/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:17.227710 118557 storage/replica_command.go:979 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] could not promote [] to voter, rolling back: change replicas of r64 failed: storage/replica_raft.go:305: [n1,s1,r64/2:/Table/78/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:17.227750 118557 storage/queue.go:1027 [n1,replicate,s1,r64/2:/Table/78/1{-/3}] change replicas of r64 failed: storage/replica_raft.go:305: [n1,s1,r64/2:/Table/78/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:17.228153 119290 storage/replica_command.go:1430 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r101:/Table/78/1/{3-4} [(n1,s1):1, next=2, gen=106] I190903 20:13:17.229364 119083 storage/replica_raft.go:291 [n3,s3,r67/2:/{Table/78/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 20:13:17.230174 119107 storage/replica_raft.go:291 [n2,s2,r66/3:/Table/78/{1/5-2}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 20:13:17.232503 119290 storage/replica_raft.go:291 [n1,s1,r101/1:/Table/78/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:17.233933 119290 storage/store_snapshot.go:995 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] sending LEARNER snapshot 48f7a144 at applied index 14 I190903 20:13:17.234406 119290 storage/store_snapshot.go:1038 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:17.236751 118911 storage/replica_raftstorage.go:793 [n3,s3,r101/2:{-}] applying LEARNER snapshot [id=48f7a144 index=14] I190903 20:13:17.241100 117806 storage/replica_command.go:609 [n2,merge,s2,r65/1:/Table/78/1/{4-5}] initiating a merge of r66:/Table/78/{1/5-2} [(n2,s2):3, next=4, gen=110] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:17.247149 118911 storage/replica_raftstorage.go:814 [n3,s3,r101/2:/Table/78/1/{3-4}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=48f7a144 index=14] I190903 20:13:17.250456 119290 storage/replica_command.go:1430 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r101:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=107] I190903 20:13:17.253799 1716 storage/store.go:2593 [n2,s2,r65/1:/Table/78/1/{4-5}] removing replica r66/3 I190903 20:13:17.254538 119326 storage/store.go:2593 [n3,replicaGC,s3,r66/2:/Table/78/{1/5-2}] removing replica r66/2 I190903 20:13:17.254627 119294 storage/store.go:2593 [n2,replicaGC,s2,r67/1:/{Table/78/2-Max}] removing replica r67/1 I190903 20:13:17.254930 119294 storage/replica_destroy.go:152 [n2,replicaGC,s2,r67/1:/{Table/78/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:17.257455 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.258884 119290 storage/replica_raft.go:291 [n1,s1,r101/1:/Table/78/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:17.259399 119290 storage/replica_command.go:1430 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r101:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=108] I190903 20:13:17.259950 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.267108 119326 storage/replica_destroy.go:152 [n3,replicaGC,s3,r66/2:/Table/78/{1/5-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:17.272340 119290 storage/replica_raft.go:291 [n1,s1,r101/1:/Table/78/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:17.272387 119290 storage/replica_raft.go:304 [n1,s1,r101/1:/Table/78/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:17.274225 119290 storage/replica_command.go:979 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r101 failed: storage/replica_raft.go:305: [n1,s1,r101/1:/Table/78/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:17.274254 119290 storage/queue.go:1027 [n1,replicate,s1,r101/1:/Table/78/1/{3-4}] change replicas of r101 failed: storage/replica_raft.go:305: [n1,s1,r101/1:/Table/78/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:17.306257 119168 storage/queue.go:1027 [n3,replicate,s3,r101/2:/Table/78/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 20:13:17.320220 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.324153 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.329774 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.336663 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.347076 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 20:13:17.349673 119610 storage/queue.go:1027 [n2,replicate,s2,r64/3:/Table/78/1{-/3}] no removable replicas from range that needs a removal: [2:0, 3*:43] I190903 20:13:17.404350 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.408995 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.418838 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.434710 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.448856 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.477551 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.562180 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.565441 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.568459 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.633592 120550 storage/replica_command.go:1430 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r65:/Table/78/{1/4-2} [(n2,s2):1, next=2, gen=111] I190903 20:13:17.642499 120550 storage/replica_raft.go:291 [n2,s2,r65/1:/Table/78/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:17.643152 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.643854 120550 storage/store_snapshot.go:995 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] sending LEARNER snapshot a7042b85 at applied index 28 I190903 20:13:17.644046 120550 storage/store_snapshot.go:1038 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 22, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:17.645284 120561 storage/replica_raftstorage.go:793 [n1,s1,r65/2:{-}] applying LEARNER snapshot [id=a7042b85 index=28] I190903 20:13:17.646627 120561 storage/replica_raftstorage.go:814 [n1,s1,r65/2:/Table/78/{1/4-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=a7042b85 index=28] I190903 20:13:17.646894 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.647400 120550 storage/replica_command.go:1430 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r65:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=112] I190903 20:13:17.649991 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.661117 120550 storage/replica_raft.go:291 [n2,s2,r65/1:/Table/78/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:17.661719 120550 storage/replica_command.go:1430 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r65:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=113] I190903 20:13:17.676747 120550 storage/replica_raft.go:291 [n2,s2,r65/1:/Table/78/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:17.676805 120550 storage/replica_raft.go:304 [n2,s2,r65/1:/Table/78/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:17.678166 120550 storage/replica_command.go:979 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r65 failed: storage/replica_raft.go:305: [n2,s2,r65/1:/Table/78/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:17.678197 120550 storage/queue.go:1027 [n2,replicate,s2,r65/1:/Table/78/{1/4-2}] change replicas of r65 failed: storage/replica_raft.go:305: [n2,s2,r65/1:/Table/78/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:17.791491 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:17.794896 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:17.798058 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:17.824706 121221 storage/replica_command.go:1430 [n3,replicate,s3,r101/2:/Table/78/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r101:/Table/78/1/{3-4} [(n1,s1):1, (n3,s3):2, next=3, gen=108] I190903 20:13:17.832756 121221 storage/replica_raft.go:291 [n3,s3,r101/2:/Table/78/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:17.840634 121252 storage/store.go:2593 [n1,replicaGC,s1,r101/1:/Table/78/1/{3-4}] removing replica r101/1 I190903 20:13:17.841031 121252 storage/replica_destroy.go:152 [n1,replicaGC,s1,r101/1:/Table/78/1/{3-4}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:17.843361 121286 storage/replica_command.go:1430 [n2,replicate,s2,r64/3:/Table/78/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r64:/Table/78/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=108] I190903 20:13:17.850575 121286 storage/replica_raft.go:291 [n2,s2,r64/3:/Table/78/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:17.861082 121231 storage/store.go:2593 [n1,replicaGC,s1,r64/2:/Table/78/1{-/3}] removing replica r64/2 I190903 20:13:17.861536 121231 storage/replica_destroy.go:152 [n1,replicaGC,s1,r64/2:/Table/78/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:18.019035 121753 storage/queue.go:1027 [n1,replicate,s1,r65/2:/Table/78/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:35] I190903 20:13:18.069634 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:18.072728 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:18.076512 111105 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 === RUN TestRepartitioning/multi_col_range_partitioning/multi_col_range_partitioning_-_MAXVALUE I190903 20:13:18.139643 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=78 (single col range partitioning) version=5 mtime=2019-09-03 20:13:18.129959672 +0000 UTC I190903 20:13:18.146233 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 77, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 20:13:18.148508 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 78 4}] I190903 20:13:18.177144 122014 storage/replica_command.go:1430 [n1,replicate,s1,r65/2:/Table/78/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r65:/Table/78/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=113] I190903 20:13:18.186013 122014 storage/replica_raft.go:291 [n1,s1,r65/2:/Table/78/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:18.194365 122599 storage/store.go:2593 [n2,replicaGC,s2,r65/1:/Table/78/{1/4-2}] removing replica r65/1 I190903 20:13:18.194888 122599 storage/replica_destroy.go:152 [n2,replicaGC,s2,r65/1:/Table/78/{1/4-2}] removed 13 (0+13) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:18.208206 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=78 (single col range partitioning) version=6 mtime=2019-09-03 20:13:18.208019663 +0000 UTC I190903 20:13:18.254130 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 79, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:18.260987 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 80, info: {TableName:data.public."multi col range partitioning" Statement:CREATE TABLE "multi col range partitioning" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p56 VALUES FROM (3, 4) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 20:13:18.263221 123442 storage/replica_command.go:284 [n3,split,s3,r67/2:/{Table/78/2-Max}] initiating a split of this range at key /Table/80 [r128] (zone config) I190903 20:13:18.270361 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:18.272096 123680 storage/replica_command.go:284 [n3,split,s3,r128/1:/{Table/80-Max}] initiating a split of this range at key /Table/80/1 [r129] (zone config) I190903 20:13:18.279662 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:18.291404 123953 storage/replica_command.go:284 [n3,split,s3,r129/1:/{Table/80/1-Max}] initiating a split of this range at key /Table/80/1/3/4 [r130] (zone config) I190903 20:13:18.292866 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:18.309207 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:18.309962 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:18.328857 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.332531 124399 storage/replica_command.go:284 [n3,split,s3,r130/1:/{Table/80/1/3…-Max}] initiating a split of this range at key /Table/80/1/5/6 [r131] (zone config) I190903 20:13:18.339705 125100 storage/replica_command.go:1430 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r129:/Table/80/1{-/3/4} [(n3,s3):1, next=2, gen=110] I190903 20:13:18.350874 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.358868 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.363807 125100 storage/replica_raft.go:291 [n3,s3,r129/1:/Table/80/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:18.372253 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.372578 125100 storage/store_snapshot.go:995 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] sending LEARNER snapshot 52953798 at applied index 17 I190903 20:13:18.373110 125100 storage/store_snapshot.go:1038 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:18.376494 125261 storage/replica_raftstorage.go:793 [n2,s2,r129/2:{-}] applying LEARNER snapshot [id=52953798 index=17] I190903 20:13:18.382379 125118 storage/replica_command.go:284 [n3,split,s3,r131/1:/{Table/80/1/5…-Max}] initiating a split of this range at key /Table/80/1/5/7 [r132] (zone config) I190903 20:13:18.385089 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.388100 125261 storage/replica_raftstorage.go:814 [n2,s2,r129/2:/Table/80/1{-/3/4}] applied LEARNER snapshot [total=12ms ingestion=4@11ms id=52953798 index=17] I190903 20:13:18.393346 125100 storage/replica_command.go:1430 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r129:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=111] I190903 20:13:18.395147 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:18.402658 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.412507 125100 storage/replica_raft.go:291 [n3,s3,r129/1:/Table/80/1{-/3/4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:18.412563 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.412861 125433 storage/replica_command.go:284 [n3,split,s3,r132/1:/{Table/80/1/5…-Max}] initiating a split of this range at key /Table/80/2 [r133] (zone config) I190903 20:13:18.413107 125100 storage/replica_command.go:1430 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r129:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=112] I190903 20:13:18.443578 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.444305 125100 storage/replica_raft.go:291 [n3,s3,r129/1:/Table/80/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:18.444363 125100 storage/replica_raft.go:304 [n3,s3,r129/1:/Table/80/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:18.445235 125100 storage/replica_command.go:979 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r129 failed: storage/replica_raft.go:305: [n3,s3,r129/1:/Table/80/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:18.445275 125100 storage/queue.go:1027 [n3,replicate,s3,r129/1:/Table/80/1{-/3/4}] change replicas of r129 failed: storage/replica_raft.go:305: [n3,s3,r129/1:/Table/80/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:18.445738 125532 storage/replica_command.go:1430 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r131:/Table/80/1/5/{6-7} [(n3,s3):1, next=2, gen=112] I190903 20:13:18.458549 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.459381 125532 storage/replica_raft.go:291 [n3,s3,r131/1:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:18.460705 125532 storage/store_snapshot.go:995 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] sending LEARNER snapshot 56b53669 at applied index 18 I190903 20:13:18.460867 125532 storage/store_snapshot.go:1038 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:18.462176 125629 storage/replica_raftstorage.go:793 [n1,s1,r131/2:{-}] applying LEARNER snapshot [id=56b53669 index=18] I190903 20:13:18.463502 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.464355 125629 storage/replica_raftstorage.go:814 [n1,s1,r131/2:/Table/80/1/5/{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=56b53669 index=18] I190903 20:13:18.466175 125532 storage/replica_command.go:1430 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r131:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113] I190903 20:13:18.483765 125532 storage/replica_raft.go:291 [n3,s3,r131/1:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:18.484309 125532 storage/replica_command.go:1430 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r131:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:18.504858 125532 storage/replica_raft.go:291 [n3,s3,r131/1:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:18.504895 125532 storage/replica_raft.go:304 [n3,s3,r131/1:/Table/80/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:18.505775 125532 storage/replica_command.go:979 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r131 failed: storage/replica_raft.go:305: [n3,s3,r131/1:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:18.505807 125532 storage/queue.go:1027 [n3,replicate,s3,r131/1:/Table/80/1/5/{6-7}] change replicas of r131 failed: storage/replica_raft.go:305: [n3,s3,r131/1:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:18.506213 125735 storage/replica_command.go:1430 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r132:/Table/80/{1/5/7-2} [(n3,s3):1, next=2, gen=113] I190903 20:13:18.506323 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:18.521762 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:18.525500 125735 storage/replica_raft.go:291 [n3,s3,r132/1:/Table/80/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:18.527538 125735 storage/store_snapshot.go:995 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] sending LEARNER snapshot 27227fea at applied index 18 I190903 20:13:18.527706 125735 storage/store_snapshot.go:1038 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:18.531766 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.532179 125187 storage/replica_raftstorage.go:793 [n1,s1,r132/2:{-}] applying LEARNER snapshot [id=27227fea index=18] I190903 20:13:18.547701 125187 storage/replica_raftstorage.go:814 [n1,s1,r132/2:/Table/80/{1/5/7-2}] applied LEARNER snapshot [total=15ms ingestion=4@15ms id=27227fea index=18] I190903 20:13:18.559898 125735 storage/replica_command.go:1430 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r132:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=114] I190903 20:13:18.570599 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.591190 125735 storage/replica_raft.go:291 [n3,s3,r132/1:/Table/80/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:18.591843 125735 storage/replica_command.go:1430 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r132:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=115] I190903 20:13:18.608819 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.618854 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.629280 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:18.643802 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:18.643803 125735 storage/replica_raft.go:291 [n3,s3,r132/1:/Table/80/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:18.644014 125735 storage/replica_raft.go:304 [n3,s3,r132/1:/Table/80/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:18.647296 125735 storage/replica_command.go:979 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r132 failed: storage/replica_raft.go:305: [n3,s3,r132/1:/Table/80/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:18.647343 125735 storage/queue.go:1027 [n3,replicate,s3,r132/1:/Table/80/{1/5/7-2}] change replicas of r132 failed: storage/replica_raft.go:305: [n3,s3,r132/1:/Table/80/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:18.657508 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:18.664469 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.677375 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.701311 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.712164 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:18.725526 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:18.750907 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 20:13:18.775290 126758 storage/queue.go:1027 [n1,replicate,s1,r132/2:/Table/80/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:18.794181 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:18.866630 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:18.883823 127025 storage/replica_command.go:1430 [n1,replicate,s1,r132/2:/Table/80/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r132:/Table/80/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=115] I190903 20:13:18.890407 127025 storage/replica_raft.go:291 [n1,s1,r132/2:/Table/80/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:18.901154 127073 storage/store.go:2593 [n3,replicaGC,s3,r132/1:/Table/80/{1/5/7-2}] removing replica r132/1 I190903 20:13:18.901575 127073 storage/replica_destroy.go:152 [n3,replicaGC,s3,r132/1:/Table/80/{1/5/7-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:18.933044 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 20:13:19.027770 127477 storage/queue.go:1027 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:19.058916 127467 storage/queue.go:1027 [n2,replicate,s2,r129/2:/Table/80/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:19.074373 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:19.083125 127555 storage/replica_command.go:1430 [n2,replicate,s2,r129/2:/Table/80/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r129:/Table/80/1{-/3/4} [(n3,s3):1, (n2,s2):2, next=3, gen=112] I190903 20:13:19.090378 127555 storage/replica_raft.go:291 [n2,s2,r129/2:/Table/80/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:19.099992 127661 storage/store.go:2593 [n3,replicaGC,s3,r129/1:/Table/80/1{-/3/4}] removing replica r129/1 I190903 20:13:19.100378 127661 storage/replica_destroy.go:152 [n3,replicaGC,s3,r129/1:/Table/80/1{-/3/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:19.272484 128126 storage/replica_command.go:1430 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r131:/Table/80/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:19.276550 128126 storage/replica_raft.go:291 [n1,s1,r131/2:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:19.285326 128181 storage/store.go:2593 [n3,replicaGC,s3,r131/1:/Table/80/1/5/{6-7}] removing replica r131/1 I190903 20:13:19.285769 128181 storage/replica_destroy.go:152 [n3,replicaGC,s3,r131/1:/Table/80/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:19.407186 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:19.409180 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:19.415505 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:19.418956 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:19.422597 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 20:13:19.42094854 +0000 UTC I190903 20:13:19.432294 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 1}] I190903 20:13:19.434873 128370 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=2:1567541903.629229165,0, refcount=0 I190903 20:13:19.489081 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=80 (multi col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 20:13:19.488919859 +0000 UTC I190903 20:13:19.495266 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 2}] I190903 20:13:19.496460 129098 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=3:1567541907.473929838,0, refcount=0 I190903 20:13:19.526422 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 20:13:19.523222775 +0000 UTC I190903 20:13:19.534330 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 80, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:19.536477 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 3}] I190903 20:13:19.545217 129443 sql/lease.go:923 new lease: 80("multi col range partitioning - MAXVALUE") ver=4:1567541911.056247647,0, refcount=0 I190903 20:13:19.595921 130160 storage/replica_command.go:284 [n3,split,s3,r130/1:/Table/80/1/{3/4-5/6}] initiating a split of this range at key /Table/80/1/4 [r134] (zone config) I190903 20:13:19.596858 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:19.607050 130289 storage/replica_command.go:1430 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r130:/Table/80/1/{3/4-4} [(n3,s3):1, next=2, gen=112] I190903 20:13:19.621707 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:19.625526 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:19.631333 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:19.638277 130644 storage/replica_command.go:284 [n2,split,s2,r129/2:/Table/80/1{-/3/4}] initiating a split of this range at key /Table/80/1/3 [r68] (zone config) I190903 20:13:19.669106 130289 storage/replica_raft.go:291 [n3,s3,r130/1:/Table/80/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:19.671313 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:19.672998 130289 storage/store_snapshot.go:995 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] sending LEARNER snapshot f967ff1b at applied index 24 I190903 20:13:19.673171 130289 storage/store_snapshot.go:1038 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:19.683391 130321 storage/replica_raftstorage.go:793 [n1,s1,r130/2:{-}] applying LEARNER snapshot [id=f967ff1b index=24] I190903 20:13:19.684694 130321 storage/replica_raftstorage.go:814 [n1,s1,r130/2:/Table/80/1/{3/4-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=f967ff1b index=24] I190903 20:13:19.685841 130289 storage/replica_command.go:1430 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113] I190903 20:13:19.690643 131149 storage/replica_command.go:1430 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r68:/Table/80/1/3{-/4} [(n2,s2):1, next=2, gen=114] I190903 20:13:19.718818 130289 storage/replica_raft.go:291 [n3,s3,r130/1:/Table/80/1/{3/4-4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:19.719960 130289 storage/replica_command.go:1430 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:19.725055 130475 storage/replica_command.go:979 [n3,s3,r130/1:/Table/80/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r130 failed: descriptor changed: [expected] r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113] != [actual] r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=114] E190903 20:13:19.725295 131148 storage/queue.go:1027 [n2,merge,s2,r68/1:/Table/80/1/3{-/4}] removing learners from r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113]: change replicas of r130 failed: descriptor changed: [expected] r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113] != [actual] r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:19.734277 131149 storage/replica_raft.go:291 [n2,s2,r68/1:/Table/80/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:19.736639 131149 storage/store_snapshot.go:995 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] sending LEARNER snapshot 87079eeb at applied index 15 I190903 20:13:19.736788 131149 storage/store_snapshot.go:1038 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:19.737970 131273 storage/replica_raftstorage.go:793 [n3,s3,r68/2:{-}] applying LEARNER snapshot [id=87079eeb index=15] I190903 20:13:19.739081 130289 storage/replica_raft.go:291 [n3,s3,r130/1:/Table/80/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:19.739117 130289 storage/replica_raft.go:304 [n3,s3,r130/1:/Table/80/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:19.739308 131273 storage/replica_raftstorage.go:814 [n3,s3,r68/2:/Table/80/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=87079eeb index=15] I190903 20:13:19.745106 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:19.752198 131149 storage/replica_command.go:1430 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=115] I190903 20:13:19.762908 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 80, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:19.763028 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.767636 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.773241 131149 storage/replica_raft.go:291 [n2,s2,r68/1:/Table/80/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:19.774059 131149 storage/replica_command.go:1430 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2, next=3, gen=116] I190903 20:13:19.786395 130289 storage/replica_command.go:979 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r130 failed: storage/replica_raft.go:305: [n3,s3,r130/1:/Table/80/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:19.786438 130289 storage/queue.go:1027 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] change replicas of r130 failed: storage/replica_raft.go:305: [n3,s3,r130/1:/Table/80/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:19.805841 131149 storage/replica_raft.go:291 [n2,s2,r68/1:/Table/80/1/3{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:19.805878 131149 storage/replica_raft.go:304 [n2,s2,r68/1:/Table/80/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:19.807463 131149 storage/replica_command.go:979 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r68 failed: storage/replica_raft.go:305: [n2,s2,r68/1:/Table/80/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:19.807496 131149 storage/queue.go:1027 [n2,replicate,s2,r68/1:/Table/80/1/3{-/4}] change replicas of r68 failed: storage/replica_raft.go:305: [n2,s2,r68/1:/Table/80/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:19.810141 132128 storage/queue.go:1027 [n3,replicate,s3,r68/2:/Table/80/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:21] I190903 20:13:19.822562 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.831278 132246 storage/replica_command.go:1430 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r131:/Table/80/1/5/{6-7} [(n1,s1):2, next=3, gen=115] I190903 20:13:19.834724 132246 storage/replica_raft.go:291 [n1,s1,r131/2:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:19.834904 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.837909 132246 storage/store_snapshot.go:995 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] sending LEARNER snapshot bb924c13 at applied index 33 I190903 20:13:19.838172 132246 storage/store_snapshot.go:1038 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:19.840250 132265 storage/replica_raftstorage.go:793 [n2,s2,r131/3:{-}] applying LEARNER snapshot [id=bb924c13 index=33] I190903 20:13:19.844572 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.847688 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.848922 132265 storage/replica_raftstorage.go:814 [n2,s2,r131/3:/Table/80/1/5/{6-7}] applied LEARNER snapshot [total=9ms ingestion=4@8ms id=bb924c13 index=33] I190903 20:13:19.849869 132246 storage/replica_command.go:1430 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r131:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=116] I190903 20:13:19.855820 132246 storage/replica_raft.go:291 [n1,s1,r131/2:/Table/80/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:19.856415 132246 storage/replica_command.go:1430 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r131:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=117] I190903 20:13:19.859882 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.866217 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.867083 132246 storage/replica_raft.go:291 [n1,s1,r131/2:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:19.867117 132246 storage/replica_raft.go:304 [n1,s1,r131/2:/Table/80/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:19.868505 132246 storage/replica_command.go:979 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r131 failed: storage/replica_raft.go:305: [n1,s1,r131/2:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:19.868547 132246 storage/queue.go:1027 [n1,replicate,s1,r131/2:/Table/80/1/5/{6-7}] change replicas of r131 failed: storage/replica_raft.go:305: [n1,s1,r131/2:/Table/80/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:19.879390 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.888036 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.895537 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.902000 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.910333 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.922610 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.929448 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.933285 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.933412 131674 storage/replica_command.go:979 [n2,s2,r68/1:/Table/80/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r68 failed: descriptor changed: [expected] r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=115] != [actual] r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2, next=3, gen=116] E190903 20:13:19.933609 131674 storage/queue.go:1027 [n2,merge,s2,r68/1:/Table/80/1/3{-/4}] removing learners from r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=115]: change replicas of r68 failed: descriptor changed: [expected] r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=115] != [actual] r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2, next=3, gen=116] I190903 20:13:19.938738 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.943478 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.948246 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.952899 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.956500 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.960224 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.963965 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.968482 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.971735 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.977748 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.981070 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:19.985682 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:19.990239 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:19.992435 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:19.997675 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.001716 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.005199 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.008260 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.012088 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.016760 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.023650 133105 storage/replica_command.go:1430 [n3,replicate,s3,r130/1:/Table/80/1/{3/4-4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r130:/Table/80/1/{3/4-4} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:20.025217 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.031157 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.032145 133105 storage/replica_raft.go:291 [n3,s3,r130/1:/Table/80/1/{3/4-4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):1] next=3 I190903 20:13:20.035760 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:20.038827 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.043630 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.046907 133175 storage/store.go:2593 [n1,replicaGC,s1,r130/2:/Table/80/1/{3/4-4}] removing replica r130/2 I190903 20:13:20.047254 133175 storage/replica_destroy.go:152 [n1,replicaGC,s1,r130/2:/Table/80/1/{3/4-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:20.048062 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.052985 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.058403 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.069829 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.077024 133343 storage/replica_command.go:1430 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r134:/Table/80/1/{4-5/6} [(n3,s3):1, next=2, gen=112] I190903 20:13:20.081549 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.092531 133343 storage/replica_raft.go:291 [n3,s3,r134/1:/Table/80/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:20.098432 133343 storage/store_snapshot.go:995 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] sending LEARNER snapshot 189abecf at applied index 15 I190903 20:13:20.098608 133343 storage/store_snapshot.go:1038 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:20.100941 133185 storage/replica_raftstorage.go:793 [n1,s1,r134/2:{-}] applying LEARNER snapshot [id=189abecf index=15] I190903 20:13:20.101894 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.102554 133185 storage/replica_raftstorage.go:814 [n1,s1,r134/2:/Table/80/1/{4-5/6}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=189abecf index=15] I190903 20:13:20.103051 133343 storage/replica_command.go:1430 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r134:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=113] I190903 20:13:20.107360 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.110912 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.111922 133343 storage/replica_raft.go:291 [n3,s3,r134/1:/Table/80/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:20.112533 133343 storage/replica_command.go:1430 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r134:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:20.121533 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.124916 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.127981 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.133592 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:20.135470 133343 storage/replica_raft.go:291 [n3,s3,r134/1:/Table/80/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:20.135504 133343 storage/replica_raft.go:304 [n3,s3,r134/1:/Table/80/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:20.143628 133343 storage/replica_command.go:979 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r134 failed: storage/replica_raft.go:305: [n3,s3,r134/1:/Table/80/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:20.143713 133343 storage/queue.go:1027 [n3,replicate,s3,r134/1:/Table/80/1/{4-5/6}] change replicas of r134 failed: storage/replica_raft.go:305: [n3,s3,r134/1:/Table/80/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:20.153387 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:20.159994 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.165686 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.173373 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.182698 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.196587 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.200646 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.206975 133845 storage/replica_command.go:1430 [n3,replicate,s3,r68/2:/Table/80/1/3{-/4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r68:/Table/80/1/3{-/4} [(n2,s2):1, (n3,s3):2, next=3, gen=116] E190903 20:13:20.207829 133832 storage/queue.go:1027 [n3,merge,s3,r68/2:/Table/80/1/3{-/4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:20.216155 133845 storage/replica_raft.go:291 [n3,s3,r68/2:/Table/80/1/3{-/4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 20:13:20.221427 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:20.227607 133895 storage/store.go:2593 [n2,replicaGC,s2,r68/1:/Table/80/1/3{-/4}] removing replica r68/1 I190903 20:13:20.227949 133895 storage/replica_destroy.go:152 [n2,replicaGC,s2,r68/1:/Table/80/1/3{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:20.258757 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.261926 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.265585 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.337297 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:20.383705 134358 storage/replica_command.go:609 [n3,merge,s3,r68/2:/Table/80/1/3{-/4}] initiating a merge of r130:/Table/80/1/{3/4-4} [(n3,s3):1, next=3, gen=115] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:20.392704 2255 storage/store.go:2593 [n3,s3,r68/2:/Table/80/1/3{-/4}] removing replica r130/1 E190903 20:13:20.454753 134513 storage/queue.go:1027 [n2,replicate,s2,r131/3:/Table/80/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:42] I190903 20:13:20.475510 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 20:13:20.549291 134856 storage/queue.go:1027 [n1,replicate,s1,r134/2:/Table/80/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 20:13:20.564332 134744 storage/replica_command.go:1430 [n2,replicate,s2,r131/3:/Table/80/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r131:/Table/80/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=117] I190903 20:13:20.571166 134744 storage/replica_raft.go:291 [n2,s2,r131/3:/Table/80/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:20.579363 134908 storage/store.go:2593 [n1,replicaGC,s1,r131/2:/Table/80/1/5/{6-7}] removing replica r131/2 I190903 20:13:20.579840 134908 storage/replica_destroy.go:152 [n1,replicaGC,s1,r131/2:/Table/80/1/5/{6-7}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:20.716656 298 server/status/runtime.go:498 [n1] runtime stats: 255 MiB RSS, 954 goroutines, 64 MiB/33 MiB/138 MiB GO alloc/idle/total, 54 MiB/107 MiB CGO alloc/total, 7482.6 CGO/sec, 73.8/8.7 %(u/s)time, 0.5 %gc (12x), 99 MiB/99 MiB (r/w)net I190903 20:13:20.797076 135411 storage/replica_command.go:1430 [n1,replicate,s1,r134/2:/Table/80/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r134:/Table/80/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=114] I190903 20:13:20.800204 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:20.803432 135411 storage/replica_raft.go:291 [n1,s1,r134/2:/Table/80/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:20.805831 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:20.809521 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:20.816109 135371 storage/store.go:2593 [n3,replicaGC,s3,r134/1:/Table/80/1/{4-5/6}] removing replica r134/1 I190903 20:13:20.816380 135371 storage/replica_destroy.go:152 [n3,replicaGC,s3,r134/1:/Table/80/1/{4-5/6}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:21.330048 1776 server/status/runtime.go:498 [n2] runtime stats: 255 MiB RSS, 951 goroutines, 77 MiB/24 MiB/138 MiB GO alloc/idle/total, 54 MiB/107 MiB CGO alloc/total, 7098.5 CGO/sec, 70.5/8.3 %(u/s)time, 0.5 %gc (11x), 98 MiB/98 MiB (r/w)net I190903 20:13:21.350960 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.353583 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:21.356637 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.358377 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:21.361681 122043 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE/multi_col_range_partitioning I190903 20:13:21.375587 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=80 (multi col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 20:13:21.363798448 +0000 UTC I190903 20:13:21.401965 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 79, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE"]} I190903 20:13:21.404120 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 80 4}] I190903 20:13:21.408817 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:21.457805 2354 server/status/runtime.go:498 [n3] runtime stats: 263 MiB RSS, 950 goroutines, 52 MiB/41 MiB/139 MiB GO alloc/idle/total, 56 MiB/111 MiB CGO alloc/total, 7168.6 CGO/sec, 71.9/8.4 %(u/s)time, 0.5 %gc (12x), 98 MiB/98 MiB (r/w)net I190903 20:13:21.461113 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=80 (multi col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 20:13:21.460937549 +0000 UTC I190903 20:13:21.503732 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 81, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:21.522281 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 82, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 20:13:21.527963 133218 storage/replica_command.go:284 [n3,split,s3,r133/1:/{Table/80/2-Max}] initiating a split of this range at key /Table/82 [r135] (zone config) I190903 20:13:21.584541 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:21.586876 137503 storage/replica_command.go:284 [n3,split,s3,r135/1:/{Table/82-Max}] initiating a split of this range at key /Table/82/1 [r136] (zone config) I190903 20:13:21.612820 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:21.618273 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:21.625118 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:21.653996 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:21.663985 140072 storage/replica_command.go:284 [n3,split,s3,r136/1:/{Table/82/1-Max}] initiating a split of this range at key /Table/82/1/3 [r137] (zone config) I190903 20:13:21.670500 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:21.766942 140304 storage/replica_command.go:284 [n3,split,s3,r137/1:/{Table/82/1/3-Max}] initiating a split of this range at key /Table/82/1/4 [r138] (zone config) I190903 20:13:21.767561 140465 storage/replica_command.go:1430 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r136:/Table/82/1{-/3} [(n3,s3):1, next=2, gen=116] I190903 20:13:21.769569 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:21.769766 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.782209 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:21.797035 140465 storage/replica_raft.go:291 [n3,s3,r136/1:/Table/82/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:21.799413 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.799836 140465 storage/store_snapshot.go:995 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] sending LEARNER snapshot 021e8e76 at applied index 17 I190903 20:13:21.799983 140465 storage/store_snapshot.go:1038 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:21.803584 141031 storage/replica_raftstorage.go:793 [n2,s2,r136/2:{-}] applying LEARNER snapshot [id=021e8e76 index=17] I190903 20:13:21.807118 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:21.809415 140976 storage/replica_command.go:284 [n3,split,s3,r138/1:/{Table/82/1/4-Max}] initiating a split of this range at key /Table/82/1/5/6 [r139] (zone config) I190903 20:13:21.820454 141031 storage/replica_raftstorage.go:814 [n2,s2,r136/2:/Table/82/1{-/3}] applied LEARNER snapshot [total=17ms ingestion=4@16ms id=021e8e76 index=17] I190903 20:13:21.820939 140465 storage/replica_command.go:1430 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r136:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=117] I190903 20:13:21.822263 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:21.831842 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:21.836369 141159 storage/replica_command.go:284 [n3,split,s3,r139/1:/{Table/82/1/5…-Max}] initiating a split of this range at key /Table/82/1/5/7 [r140] (zone config) I190903 20:13:21.837910 140465 storage/replica_raft.go:291 [n3,s3,r136/1:/Table/82/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:21.838628 140465 storage/replica_command.go:1430 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r136:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=118] I190903 20:13:21.841498 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.856565 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.862373 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.868307 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.871018 141365 storage/replica_command.go:284 [n3,split,s3,r140/1:/{Table/82/1/5…-Max}] initiating a split of this range at key /Table/82/2 [r151] (zone config) I190903 20:13:21.874508 140465 storage/replica_raft.go:291 [n3,s3,r136/1:/Table/82/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:21.874550 140465 storage/replica_raft.go:304 [n3,s3,r136/1:/Table/82/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:21.876096 140465 storage/replica_command.go:979 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] could not promote [] to voter, rolling back: change replicas of r136 failed: storage/replica_raft.go:305: [n3,s3,r136/1:/Table/82/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:21.876267 140465 storage/queue.go:1027 [n3,replicate,s3,r136/1:/Table/82/1{-/3}] change replicas of r136 failed: storage/replica_raft.go:305: [n3,s3,r136/1:/Table/82/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:21.878162 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.880539 140961 storage/replica_command.go:1430 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r138:/Table/82/1/{4-5/6} [(n3,s3):1, next=2, gen=118] I190903 20:13:21.895339 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:21.896475 140961 storage/replica_raft.go:291 [n3,s3,r138/1:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:21.898397 140961 storage/store_snapshot.go:995 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] sending LEARNER snapshot 2a781194 at applied index 18 I190903 20:13:21.898716 140961 storage/store_snapshot.go:1038 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:21.899465 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.899972 141524 storage/replica_raftstorage.go:793 [n1,s1,r138/2:{-}] applying LEARNER snapshot [id=2a781194 index=18] I190903 20:13:21.903839 141524 storage/replica_raftstorage.go:814 [n1,s1,r138/2:/Table/82/1/{4-5/6}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=2a781194 index=18] I190903 20:13:21.904663 140961 storage/replica_command.go:1430 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r138:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] I190903 20:13:21.909959 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.916068 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:21.917408 140961 storage/replica_raft.go:291 [n3,s3,r138/1:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:21.920194 140961 storage/replica_command.go:1430 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r138:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:21.942151 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.949877 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.954356 140961 storage/replica_raft.go:291 [n3,s3,r138/1:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:21.954420 140961 storage/replica_raft.go:304 [n3,s3,r138/1:/Table/82/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:21.954735 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.955759 140961 storage/replica_command.go:979 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r138 failed: storage/replica_raft.go:305: [n3,s3,r138/1:/Table/82/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:21.955807 140961 storage/queue.go:1027 [n3,replicate,s3,r138/1:/Table/82/1/{4-5/6}] change replicas of r138 failed: storage/replica_raft.go:305: [n3,s3,r138/1:/Table/82/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:21.956355 141649 storage/replica_command.go:1430 [n3,replicate,s3,r23/2:/Table/56{-/1}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r23:/Table/56{-/1} [(n3,s3):2, next=3, gen=12] I190903 20:13:21.966486 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.971205 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:21.979958 141649 storage/replica_raft.go:291 [n3,s3,r23/2:/Table/56{-/1}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:21.980791 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.983206 141649 storage/store_snapshot.go:995 [n3,replicate,s3,r23/2:/Table/56{-/1}] sending LEARNER snapshot 93b884ac at applied index 37 I190903 20:13:21.983377 141649 storage/store_snapshot.go:1038 [n3,replicate,s3,r23/2:/Table/56{-/1}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:21.990025 141552 storage/replica_raftstorage.go:793 [n1,s1,r23/3:{-}] applying LEARNER snapshot [id=93b884ac index=37] I190903 20:13:21.992365 141552 storage/replica_raftstorage.go:814 [n1,s1,r23/3:/Table/56{-/1}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=93b884ac index=37] I190903 20:13:21.992901 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:21.993161 141649 storage/replica_command.go:1430 [n3,replicate,s3,r23/2:/Table/56{-/1}] change replicas (add [(n1,s1):3] remove []): existing descriptor r23:/Table/56{-/1} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=13] I190903 20:13:22.003280 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:22.007495 141649 storage/replica_raft.go:291 [n3,s3,r23/2:/Table/56{-/1}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:22.008043 141649 storage/replica_command.go:1430 [n3,replicate,s3,r23/2:/Table/56{-/1}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r23:/Table/56{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=14] I190903 20:13:22.017535 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:22.025400 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:22.030457 141649 storage/replica_raft.go:291 [n3,s3,r23/2:/Table/56{-/1}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:22.030541 141649 storage/replica_raft.go:304 [n3,s3,r23/2:/Table/56{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:22.032405 141649 storage/replica_command.go:979 [n3,replicate,s3,r23/2:/Table/56{-/1}] could not promote [] to voter, rolling back: change replicas of r23 failed: storage/replica_raft.go:305: [n3,s3,r23/2:/Table/56{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:22.032439 141649 storage/queue.go:1027 [n3,replicate,s3,r23/2:/Table/56{-/1}] change replicas of r23 failed: storage/replica_raft.go:305: [n3,s3,r23/2:/Table/56{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:22.032887 141969 storage/replica_command.go:1430 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r139:/Table/82/1/5/{6-7} [(n3,s3):1, next=2, gen=119] I190903 20:13:22.044606 141969 storage/replica_raft.go:291 [n3,s3,r139/1:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:22.045428 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.045945 141969 storage/store_snapshot.go:995 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] sending LEARNER snapshot 5e3d7420 at applied index 18 I190903 20:13:22.046214 141969 storage/store_snapshot.go:1038 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:22.047453 142022 storage/replica_raftstorage.go:793 [n2,s2,r139/2:{-}] applying LEARNER snapshot [id=5e3d7420 index=18] I190903 20:13:22.048793 142022 storage/replica_raftstorage.go:814 [n2,s2,r139/2:/Table/82/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5e3d7420 index=18] I190903 20:13:22.050488 141969 storage/replica_command.go:1430 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r139:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=120] I190903 20:13:22.057254 141969 storage/replica_raft.go:291 [n3,s3,r139/1:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:22.057824 141969 storage/replica_command.go:1430 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r139:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=121] I190903 20:13:22.071354 141969 storage/replica_raft.go:291 [n3,s3,r139/1:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:22.071423 141969 storage/replica_raft.go:304 [n3,s3,r139/1:/Table/82/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:22.072088 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.072359 141969 storage/replica_command.go:979 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r139 failed: storage/replica_raft.go:305: [n3,s3,r139/1:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:22.072639 141969 storage/queue.go:1027 [n3,replicate,s3,r139/1:/Table/82/1/5/{6-7}] change replicas of r139 failed: storage/replica_raft.go:305: [n3,s3,r139/1:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:22.073454 142107 storage/replica_command.go:1430 [n3,replicate,s3,r67/2:/Table/{78/2-80}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r67:/Table/{78/2-80} [(n3,s3):2, next=3, gen=108] I190903 20:13:22.090266 142107 storage/replica_raft.go:291 [n3,s3,r67/2:/Table/{78/2-80}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:13:22.092393 142107 storage/store_snapshot.go:995 [n3,replicate,s3,r67/2:/Table/{78/2-80}] sending LEARNER snapshot 02a1e66d at applied index 36 I190903 20:13:22.092672 142107 storage/store_snapshot.go:1038 [n3,replicate,s3,r67/2:/Table/{78/2-80}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:22.094569 142290 storage/replica_raftstorage.go:793 [n2,s2,r67/3:{-}] applying LEARNER snapshot [id=02a1e66d index=36] I190903 20:13:22.112056 142290 storage/replica_raftstorage.go:814 [n2,s2,r67/3:/Table/{78/2-80}] applied LEARNER snapshot [total=17ms ingestion=4@17ms id=02a1e66d index=36] I190903 20:13:22.113047 142107 storage/replica_command.go:1430 [n3,replicate,s3,r67/2:/Table/{78/2-80}] change replicas (add [(n2,s2):3] remove []): existing descriptor r67:/Table/{78/2-80} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=109] I190903 20:13:22.113789 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.137105 142107 storage/replica_raft.go:291 [n3,s3,r67/2:/Table/{78/2-80}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 20:13:22.138111 142107 storage/replica_command.go:1430 [n3,replicate,s3,r67/2:/Table/{78/2-80}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r67:/Table/{78/2-80} [(n3,s3):2, (n2,s2):3, next=4, gen=110] I190903 20:13:22.161169 142107 storage/replica_raft.go:291 [n3,s3,r67/2:/Table/{78/2-80}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 E190903 20:13:22.161229 142107 storage/replica_raft.go:304 [n3,s3,r67/2:/Table/{78/2-80}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:22.163001 142107 storage/replica_command.go:979 [n3,replicate,s3,r67/2:/Table/{78/2-80}] could not promote [] to voter, rolling back: change replicas of r67 failed: storage/replica_raft.go:305: [n3,s3,r67/2:/Table/{78/2-80}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:22.163644 142107 storage/queue.go:1027 [n3,replicate,s3,r67/2:/Table/{78/2-80}] change replicas of r67 failed: storage/replica_raft.go:305: [n3,s3,r67/2:/Table/{78/2-80}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:22.165633 142432 storage/replica_command.go:1430 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r140:/Table/82/{1/5/7-2} [(n3,s3):1, next=2, gen=120] E190903 20:13:22.168533 142470 storage/queue.go:1027 [n1,replicate,s1,r138/2:/Table/82/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:22.183981 142432 storage/replica_raft.go:291 [n3,s3,r140/1:/Table/82/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:22.189434 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.212677 142432 storage/store_snapshot.go:995 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] sending LEARNER snapshot e506a540 at applied index 18 I190903 20:13:22.212927 142432 storage/store_snapshot.go:1038 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:22.215906 142693 storage/replica_raftstorage.go:793 [n1,s1,r140/2:{-}] applying LEARNER snapshot [id=e506a540 index=18] I190903 20:13:22.217160 142693 storage/replica_raftstorage.go:814 [n1,s1,r140/2:/Table/82/{1/5/7-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e506a540 index=18] I190903 20:13:22.217945 142432 storage/replica_command.go:1430 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r140:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=121] I190903 20:13:22.230932 142432 storage/replica_raft.go:291 [n3,s3,r140/1:/Table/82/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:22.231665 142432 storage/replica_command.go:1430 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r140:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=122] I190903 20:13:22.249722 142432 storage/replica_raft.go:291 [n3,s3,r140/1:/Table/82/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:22.249824 142432 storage/replica_raft.go:304 [n3,s3,r140/1:/Table/82/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:22.251644 142432 storage/replica_command.go:979 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r140 failed: storage/replica_raft.go:305: [n3,s3,r140/1:/Table/82/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:22.251707 142432 storage/queue.go:1027 [n3,replicate,s3,r140/1:/Table/82/{1/5/7-2}] change replicas of r140 failed: storage/replica_raft.go:305: [n3,s3,r140/1:/Table/82/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:22.254906 142813 storage/queue.go:1027 [n1,replicate,s1,r140/2:/Table/82/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:22.347502 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) E190903 20:13:22.382551 143121 storage/queue.go:1027 [n1,replicate,s1,r23/3:/Table/56{-/1}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 20:13:22.406867 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.459239 143386 storage/replica_command.go:1430 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r139:/Table/82/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=121] I190903 20:13:22.467419 143386 storage/replica_raft.go:291 [n2,s2,r139/2:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:22.486680 143420 storage/store.go:2593 [n3,replicaGC,s3,r139/1:/Table/82/1/5/{6-7}] removing replica r139/1 I190903 20:13:22.487125 143420 storage/replica_destroy.go:152 [n3,replicaGC,s3,r139/1:/Table/82/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:22.522406 143444 storage/replica_command.go:1430 [n2,replicate,s2,r67/3:/Table/{78/2-80}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r67:/Table/{78/2-80} [(n3,s3):2, (n2,s2):3, next=4, gen=110] I190903 20:13:22.530461 143444 storage/replica_raft.go:291 [n2,s2,r67/3:/Table/{78/2-80}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):3] next=4 I190903 20:13:22.540409 143653 storage/replica_command.go:1430 [n1,replicate,s1,r138/2:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r138:/Table/82/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:22.545051 143653 storage/replica_raft.go:291 [n1,s1,r138/2:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:22.559513 143752 storage/queue.go:1027 [n2,replicate,s2,r136/2:/Table/82/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:22.583165 143799 storage/replica_command.go:1430 [n1,replicate,s1,r140/2:/Table/82/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r140:/Table/82/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=122] I190903 20:13:22.588383 143799 storage/replica_raft.go:291 [n1,s1,r140/2:/Table/82/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:22.596015 143491 storage/store.go:2593 [n3,replicaGC,s3,r67/2:/Table/{78/2-80}] removing replica r67/2 I190903 20:13:22.596526 143491 storage/replica_destroy.go:152 [n3,replicaGC,s3,r67/2:/Table/{78/2-80}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:22.597726 143877 storage/store.go:2593 [n3,replicaGC,s3,r138/1:/Table/82/1/{4-5/6}] removing replica r138/1 I190903 20:13:22.598077 143877 storage/replica_destroy.go:152 [n3,replicaGC,s3,r138/1:/Table/82/1/{4-5/6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:22.647636 143840 storage/store.go:2593 [n3,replicaGC,s3,r140/1:/Table/82/{1/5/7-2}] removing replica r140/1 I190903 20:13:22.648144 143840 storage/replica_destroy.go:152 [n3,replicaGC,s3,r140/1:/Table/82/{1/5/7-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:22.678732 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:22.680725 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:22.835564 144393 storage/replica_command.go:1430 [n1,replicate,s1,r23/3:/Table/56{-/1}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r23:/Table/56{-/1} [(n3,s3):2, (n1,s1):3, next=4, gen=14] I190903 20:13:22.837802 164 storage/queue.go:518 [n1,s1,r9/1:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:22.838644 144393 storage/replica_raft.go:291 [n1,s1,r23/3:/Table/56{-/1}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:22.848619 144448 storage/store.go:2593 [n3,replicaGC,s3,r23/2:/Table/56{-/1}] removing replica r23/2 I190903 20:13:22.849061 144448 storage/replica_destroy.go:152 [n3,replicaGC,s3,r23/2:/Table/56{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:23.064632 145158 storage/replica_command.go:1430 [n2,replicate,s2,r136/2:/Table/82/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r136:/Table/82/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=118] I190903 20:13:23.071503 145158 storage/replica_raft.go:291 [n2,s2,r136/2:/Table/82/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:23.080714 145178 storage/store.go:2593 [n3,replicaGC,s3,r136/1:/Table/82/1{-/3}] removing replica r136/1 I190903 20:13:23.081087 145178 storage/replica_destroy.go:152 [n3,replicaGC,s3,r136/1:/Table/82/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:23.278032 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:23.281858 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:24.288842 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:24.292227 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:24.295885 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:24.297803 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.300904 293 storage/compactor/compactor.go:370 [n1,s1,compactor] processing compaction #1/14 (/System/tsd-/System/"tse") for 277 KiB (reasons: size=false used=true avail=false) I190903 20:13:24.301237 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:24.305013 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=82 (multi col range partitioning) version=2 mtime=2019-09-03 20:13:24.303375408 +0000 UTC I190903 20:13:24.322489 293 storage/compactor/compactor.go:386 [n1,s1,compactor] processed compaction #1/14 (/System/tsd-/System/"tse") for 277 KiB in 0.0s I190903 20:13:24.370603 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 1}] I190903 20:13:24.371001 148605 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=2:1567541919.151546634,0, refcount=0 I190903 20:13:24.419134 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=82 (multi col range partitioning) version=3 mtime=2019-09-03 20:13:24.418930849 +0000 UTC I190903 20:13:24.428854 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 2}] I190903 20:13:24.430963 148968 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=3:1567541919.152546634,0, refcount=0 I190903 20:13:24.455273 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=82 (multi col range partitioning) version=4 mtime=2019-09-03 20:13:24.453150545 +0000 UTC I190903 20:13:24.468546 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 82, info: {TableName:data.public."multi col range partitioning" Statement:ALTER TABLE "multi col range partitioning" PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p56 VALUES FROM (3, 4) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:24.497945 148975 sql/lease.go:923 new lease: 82("multi col range partitioning") ver=4:1567541919.153546634,0, refcount=0 E190903 20:13:24.498830 149762 storage/queue.go:1027 [n2,merge,s2,r136/2:/Table/82/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:24.509236 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 3}] I190903 20:13:24.554530 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:24.559202 150452 storage/replica_command.go:284 [n3,split,s3,r137/1:/Table/82/1/{3-4}] initiating a split of this range at key /Table/82/1/3/4 [r152] (zone config) I190903 20:13:24.580211 150607 storage/replica_command.go:1430 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r137:/Table/82/1/3{-/4} [(n3,s3):1, next=2, gen=118] E190903 20:13:24.581728 150586 storage/queue.go:1027 [n2,merge,s2,r136/2:/Table/82/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 E190903 20:13:24.581880 150239 storage/queue.go:1027 [n3,merge,s3,r152/1:/Table/82/1/{3/4-4}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:24.586283 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} E190903 20:13:24.592762 150900 storage/queue.go:1027 [n2,merge,s2,r136/2:/Table/82/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:24.601860 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:24.612363 150497 storage/replica_command.go:1430 [n1,s1,r138/2:/Table/82/1/{4-5/6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r138:/Table/82/1/{4-5/6} [(n1,s1):2, next=3, gen=121] I190903 20:13:24.617054 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 82, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:24.617340 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.619673 150497 storage/replica_raft.go:291 [n1,s1,r138/2:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:24.621794 150607 storage/replica_raft.go:291 [n3,s3,r137/1:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:24.624472 150607 storage/store_snapshot.go:995 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] sending LEARNER snapshot 11a1acc4 at applied index 23 I190903 20:13:24.624690 150607 storage/store_snapshot.go:1038 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:24.626309 151714 storage/replica_raftstorage.go:793 [n1,s1,r137/2:{-}] applying LEARNER snapshot [id=11a1acc4 index=23] I190903 20:13:24.627649 151714 storage/replica_raftstorage.go:814 [n1,s1,r137/2:/Table/82/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=11a1acc4 index=23] I190903 20:13:24.628159 150607 storage/replica_command.go:1430 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] E190903 20:13:24.628196 151130 storage/queue.go:1027 [n2,merge,s2,r136/2:/Table/82/1{-/3}] while adding target n2,s2: change replicas of r137 failed: descriptor changed: [expected] r137:/Table/82/1/3{-/4} [(n3,s3):1, next=2, gen=118] != [actual] r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] I190903 20:13:24.628423 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.629724 150497 storage/store_snapshot.go:995 [n1,s1,r138/2:/Table/82/1/{4-5/6}] sending LEARNER snapshot 65e2a54b at applied index 35 I190903 20:13:24.630144 150497 storage/store_snapshot.go:1038 [n1,s1,r138/2:/Table/82/1/{4-5/6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:24.632389 151765 storage/replica_raftstorage.go:793 [n3,s3,r138/3:{-}] applying LEARNER snapshot [id=65e2a54b index=35] I190903 20:13:24.638874 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.639290 150607 storage/replica_raft.go:291 [n3,s3,r137/1:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:24.639707 150607 storage/replica_command.go:1430 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:24.644837 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.646148 151768 storage/replica_command.go:979 [n3,s3,r137/1:/Table/82/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r137 failed: descriptor changed: [expected] r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] != [actual] r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=120] E190903 20:13:24.647288 151263 storage/queue.go:1027 [n2,merge,s2,r136/2:/Table/82/1{-/3}] removing learners from r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119]: change replicas of r137 failed: descriptor changed: [expected] r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=119] != [actual] r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:24.650728 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.657699 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.663806 150607 storage/replica_raft.go:291 [n3,s3,r137/1:/Table/82/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:24.663851 150607 storage/replica_raft.go:304 [n3,s3,r137/1:/Table/82/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:24.665765 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.669572 150607 storage/replica_command.go:979 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r137 failed: storage/replica_raft.go:305: [n3,s3,r137/1:/Table/82/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:24.669648 150607 storage/queue.go:1027 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] change replicas of r137 failed: storage/replica_raft.go:305: [n3,s3,r137/1:/Table/82/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:24.678464 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.686083 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.686129 151765 storage/replica_raftstorage.go:814 [n3,s3,r138/3:/Table/82/1/{4-5/6}] applied LEARNER snapshot [total=54ms ingestion=4@53ms id=65e2a54b index=35] I190903 20:13:24.686994 150497 storage/replica_command.go:1430 [n1,s1,r138/2:/Table/82/1/{4-5/6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r138:/Table/82/1/{4-5/6} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=122] I190903 20:13:24.691347 150497 storage/replica_raft.go:291 [n1,s1,r138/2:/Table/82/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 20:13:24.700805 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 20:13:24.704545 152062 storage/queue.go:1027 [n3,replicate,s3,r138/3:/Table/82/1/{4-5/6}] no removable replicas from range that needs a removal: [2:0, 3*:39] I190903 20:13:24.713345 152150 storage/replica_command.go:1430 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r139:/Table/82/1/5/{6-7} [(n2,s2):2, next=3, gen=122] I190903 20:13:24.714683 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.720927 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.727039 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.727189 152150 storage/replica_raft.go:291 [n2,s2,r139/2:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:24.729812 152150 storage/store_snapshot.go:995 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] sending LEARNER snapshot 6a45d05b at applied index 38 I190903 20:13:24.729996 152150 storage/store_snapshot.go:1038 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:24.731295 152245 storage/replica_raftstorage.go:793 [n1,s1,r139/3:{-}] applying LEARNER snapshot [id=6a45d05b index=38] I190903 20:13:24.732005 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.732719 152245 storage/replica_raftstorage.go:814 [n1,s1,r139/3:/Table/82/1/5/{6-7}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=6a45d05b index=38] I190903 20:13:24.733890 152150 storage/replica_command.go:1430 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r139:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=123] E190903 20:13:24.735085 152263 storage/queue.go:1027 [n1,replicate,s1,r137/2:/Table/82/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 20:13:24.746721 152150 storage/replica_raft.go:291 [n2,s2,r139/2:/Table/82/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:24.747607 152150 storage/replica_command.go:1430 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r139:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=124] I190903 20:13:24.779562 152150 storage/replica_raft.go:291 [n2,s2,r139/2:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:24.779645 152150 storage/replica_raft.go:304 [n2,s2,r139/2:/Table/82/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:24.781279 152150 storage/replica_command.go:979 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r139 failed: storage/replica_raft.go:305: [n2,s2,r139/2:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:24.781361 152150 storage/queue.go:1027 [n2,replicate,s2,r139/2:/Table/82/1/5/{6-7}] change replicas of r139 failed: storage/replica_raft.go:305: [n2,s2,r139/2:/Table/82/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:24.787211 152044 storage/queue.go:1027 [n1,replicate,s1,r139/3:/Table/82/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 20:13:24.795007 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.812726 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.820192 152249 storage/replica_command.go:1430 [n3,s3,r138/3:/Table/82/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r138:/Table/82/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=123] I190903 20:13:24.828604 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.840693 152249 storage/replica_raft.go:291 [n3,s3,r138/3:/Table/82/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 20:13:24.841354 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.845040 150862 storage/replica_command.go:609 [n3,merge,s3,r152/1:/Table/82/1/{3/4-4}] initiating a merge of r138:/Table/82/1/{4-5/6} [(n3,s3):3, next=4, gen=124] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:24.845326 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.847759 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:24.852308 152658 storage/store.go:2593 [n1,replicaGC,s1,r138/2:/Table/82/1/{4-5/6}] removing replica r138/2 I190903 20:13:24.852649 152658 storage/replica_destroy.go:152 [n1,replicaGC,s1,r138/2:/Table/82/1/{4-5/6}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:24.853004 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.859686 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.860026 2317 storage/store.go:2593 [n3,s3,r152/1:/Table/82/1/{3/4-4}] removing replica r138/3 I190903 20:13:24.861776 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:24.865050 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.873732 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.927925 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.932988 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.938875 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:24.942416 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:24.944289 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:24.946214 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.958978 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:24.973157 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 20:13:24.994747 153245 storage/queue.go:1027 [n3,replicate,s3,r137/1:/Table/82/1/3{-/4}] no removable replicas from range that needs a removal: [1*:33, 2:0] I190903 20:13:24.999077 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:25.037373 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:25.110074 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:25.112059 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) E190903 20:13:25.207276 153830 storage/queue.go:1027 [n1,replicate,s1,r137/2:/Table/82/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:36] I190903 20:13:25.250342 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:25.253722 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:25.255654 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) I190903 20:13:25.258452 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) W190903 20:13:25.283881 154050 storage/replica_command.go:1917 [n1,s1,r137/2:/Table/82/1/3{-/4}] while transferring lease: unable to find store 2 in range r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:25.284248 154050 storage/replica_command.go:1430 [n1,s1,r137/2:/Table/82/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r137:/Table/82/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=120] I190903 20:13:25.287942 154050 storage/replica_raft.go:291 [n1,s1,r137/2:/Table/82/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:25.288964 154050 storage/replica_command.go:1430 [n1,s1,r137/2:/Table/82/1/3{-/4}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r137:/Table/82/1/3{-/4} [(n1,s1):2, next=3, gen=121] I190903 20:13:25.296691 154096 storage/store.go:2593 [n3,replicaGC,s3,r137/1:/Table/82/1/3{-/4}] removing replica r137/1 I190903 20:13:25.296992 154096 storage/replica_destroy.go:152 [n3,replicaGC,s3,r137/1:/Table/82/1/3{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:25.299417 154050 storage/replica_raft.go:291 [n1,s1,r137/2:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:25.300842 154050 storage/store_snapshot.go:995 [n1,s1,r137/2:/Table/82/1/3{-/4}] sending LEARNER snapshot 72fbb4cf at applied index 40 I190903 20:13:25.301044 154050 storage/store_snapshot.go:1038 [n1,s1,r137/2:/Table/82/1/3{-/4}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:25.302204 154134 storage/replica_raftstorage.go:793 [n2,s2,r137/3:{-}] applying LEARNER snapshot [id=72fbb4cf index=40] I190903 20:13:25.303520 154134 storage/replica_raftstorage.go:814 [n2,s2,r137/3:/Table/82/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=72fbb4cf index=40] I190903 20:13:25.304005 154050 storage/replica_command.go:1430 [n1,s1,r137/2:/Table/82/1/3{-/4}] change replicas (add [(n2,s2):3] remove []): existing descriptor r137:/Table/82/1/3{-/4} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=122] I190903 20:13:25.307768 154050 storage/replica_raft.go:291 [n1,s1,r137/2:/Table/82/1/3{-/4}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 E190903 20:13:25.310952 152744 storage/queue.go:1027 [n2,replicate,s2,r137/3:/Table/82/1/3{-/4}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 20:13:25.362118 154234 storage/replica_command.go:1430 [n1,replicate,s1,r139/3:/Table/82/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r139:/Table/82/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=124] I190903 20:13:25.366180 154234 storage/replica_raft.go:291 [n1,s1,r139/3:/Table/82/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:25.374693 154344 storage/store.go:2593 [n2,replicaGC,s2,r139/2:/Table/82/1/5/{6-7}] removing replica r139/2 I190903 20:13:25.375164 154344 storage/replica_destroy.go:152 [n2,replicaGC,s2,r139/2:/Table/82/1/5/{6-7}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:25.409430 154251 storage/replica_command.go:1430 [n2,s2,r137/3:/Table/82/1/3{-/4}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r137:/Table/82/1/3{-/4} [(n1,s1):2, (n2,s2):3, next=4, gen=123] I190903 20:13:25.416491 154251 storage/replica_raft.go:291 [n2,s2,r137/3:/Table/82/1/3{-/4}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:25.419035 153917 storage/replica_command.go:609 [n2,merge,s2,r136/2:/Table/82/1{-/3}] initiating a merge of r137:/Table/82/1/3{-/4} [(n2,s2):3, next=4, gen=124] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:25.424157 154428 storage/store.go:2593 [n1,replicaGC,s1,r137/2:/Table/82/1/3{-/4}] removing replica r137/2 I190903 20:13:25.424717 154428 storage/replica_destroy.go:152 [n1,replicaGC,s1,r137/2:/Table/82/1/3{-/4}] removed 12 (0+12) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:25.426956 1665 storage/store.go:2593 [n2,s2,r136/2:/Table/82/1{-/3}] removing replica r137/3 I190903 20:13:25.534327 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:26.131692 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) < (3, 4) I190903 20:13:26.135668 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (3, 4) AND (a, b) < (5, 6) I190903 20:13:26.139109 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:26.141010 137091 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning" WHERE (a, b) >= (5, 7) === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE/multi_col_range_partitioning_-_MAXVALUE_MAXVALUE I190903 20:13:26.210790 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=82 (multi col range partitioning) version=5 mtime=2019-09-03 20:13:26.14297737 +0000 UTC I190903 20:13:26.220900 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 81, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning"]} I190903 20:13:26.230685 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning 82 4}] I190903 20:13:26.285708 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=82 (multi col range partitioning) version=6 mtime=2019-09-03 20:13:26.285517036 +0000 UTC I190903 20:13:26.339719 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 83, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:26.364781 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 84, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root} I190903 20:13:26.367705 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:26.369020 158308 storage/replica_command.go:284 [n3,split,s3,r151/1:/{Table/82/2-Max}] initiating a split of this range at key /Table/84 [r153] (zone config) I190903 20:13:26.382260 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:26.396550 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:26.398745 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:26.409699 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:26.424430 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:26.448863 159898 storage/replica_command.go:284 [n3,split,s3,r153/1:/{Table/84-Max}] initiating a split of this range at key /Table/84/1 [r154] (zone config) I190903 20:13:26.454983 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:26.455187 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.468198 160199 storage/replica_command.go:284 [n3,split,s3,r154/1:/{Table/84/1-Max}] initiating a split of this range at key /Table/84/1/3 [r155] (zone config) I190903 20:13:26.474853 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.482282 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.488411 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.494382 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:26.513634 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.521127 160460 storage/replica_command.go:284 [n3,split,s3,r155/1:/{Table/84/1/3-Max}] initiating a split of this range at key /Table/84/1/4 [r156] (zone config) I190903 20:13:26.523196 160489 storage/replica_command.go:1430 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r154:/Table/84/1{-/3} [(n3,s3):1, next=2, gen=123] I190903 20:13:26.526794 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:26.530157 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:26.536276 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.537607 160489 storage/replica_raft.go:291 [n3,s3,r154/1:/Table/84/1{-/3}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:26.542205 160489 storage/store_snapshot.go:995 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] sending LEARNER snapshot ed2f4ff3 at applied index 17 I190903 20:13:26.542368 160489 storage/store_snapshot.go:1038 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:26.543547 160543 storage/replica_raftstorage.go:793 [n2,s2,r154/2:{-}] applying LEARNER snapshot [id=ed2f4ff3 index=17] I190903 20:13:26.561964 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.562704 160543 storage/replica_raftstorage.go:814 [n2,s2,r154/2:/Table/84/1{-/3}] applied LEARNER snapshot [total=19ms ingestion=4@18ms id=ed2f4ff3 index=17] I190903 20:13:26.563198 160489 storage/replica_command.go:1430 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] change replicas (add [(n2,s2):2] remove []): existing descriptor r154:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=124] I190903 20:13:26.564251 160623 storage/replica_command.go:284 [n3,split,s3,r156/1:/{Table/84/1/4-Max}] initiating a split of this range at key /Table/84/1/5/6 [r157] (zone config) I190903 20:13:26.568016 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:26.576055 160489 storage/replica_raft.go:291 [n3,s3,r154/1:/Table/84/1{-/3}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:26.576594 160489 storage/replica_command.go:1430 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r154:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=125] I190903 20:13:26.578173 160709 storage/replica_command.go:284 [n3,split,s3,r157/1:/{Table/84/1/5…-Max}] initiating a split of this range at key /Table/84/1/5/7 [r158] (zone config) I190903 20:13:26.582178 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.597928 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:26.619971 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.627107 160489 storage/replica_raft.go:291 [n3,s3,r154/1:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:26.627154 160489 storage/replica_raft.go:304 [n3,s3,r154/1:/Table/84/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:26.627972 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:26.627997 160489 storage/replica_command.go:979 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] could not promote [] to voter, rolling back: change replicas of r154 failed: storage/replica_raft.go:305: [n3,s3,r154/1:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:26.628118 160489 storage/queue.go:1027 [n3,replicate,s3,r154/1:/Table/84/1{-/3}] change replicas of r154 failed: storage/replica_raft.go:305: [n3,s3,r154/1:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:26.628753 160670 storage/replica_command.go:1430 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r156:/Table/84/1/{4-5/6} [(n3,s3):1, next=2, gen=125] I190903 20:13:26.639843 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.646545 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:26.648393 161000 storage/replica_command.go:284 [n3,split,s3,r158/1:/{Table/84/1/5…-Max}] initiating a split of this range at key /Table/84/2 [r159] (zone config) I190903 20:13:26.652910 160670 storage/replica_raft.go:291 [n3,s3,r156/1:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:26.655635 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.655826 160670 storage/store_snapshot.go:995 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] sending LEARNER snapshot 9e273315 at applied index 18 I190903 20:13:26.655992 160670 storage/store_snapshot.go:1038 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:26.659592 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:26.660197 161044 storage/replica_raftstorage.go:793 [n1,s1,r156/2:{-}] applying LEARNER snapshot [id=9e273315 index=18] I190903 20:13:26.661912 161044 storage/replica_raftstorage.go:814 [n1,s1,r156/2:/Table/84/1/{4-5/6}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=9e273315 index=18] I190903 20:13:26.662909 160670 storage/replica_command.go:1430 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r156:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=126] I190903 20:13:26.713446 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:26.759814 160670 storage/replica_raft.go:291 [n3,s3,r156/1:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:26.760404 160670 storage/replica_command.go:1430 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r156:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:26.764292 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.773615 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:26.787413 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.793090 160670 storage/replica_raft.go:291 [n3,s3,r156/1:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:26.793190 160670 storage/replica_raft.go:304 [n3,s3,r156/1:/Table/84/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:26.799064 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.809949 160670 storage/replica_command.go:979 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r156 failed: storage/replica_raft.go:305: [n3,s3,r156/1:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:26.809984 160670 storage/queue.go:1027 [n3,replicate,s3,r156/1:/Table/84/1/{4-5/6}] change replicas of r156 failed: storage/replica_raft.go:305: [n3,s3,r156/1:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:26.818473 161363 storage/replica_command.go:1430 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r157:/Table/84/1/5/{6-7} [(n3,s3):1, next=2, gen=126] I190903 20:13:26.823421 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.828175 161363 storage/replica_raft.go:291 [n3,s3,r157/1:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:26.830198 161363 storage/store_snapshot.go:995 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] sending LEARNER snapshot 5474741a at applied index 18 I190903 20:13:26.830367 161363 storage/store_snapshot.go:1038 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:26.831499 161591 storage/replica_raftstorage.go:793 [n2,s2,r157/2:{-}] applying LEARNER snapshot [id=5474741a index=18] I190903 20:13:26.840402 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.850223 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.850622 161591 storage/replica_raftstorage.go:814 [n2,s2,r157/2:/Table/84/1/5/{6-7}] applied LEARNER snapshot [total=19ms ingestion=4@18ms id=5474741a index=18] I190903 20:13:26.854172 161363 storage/replica_command.go:1430 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] change replicas (add [(n2,s2):2] remove []): existing descriptor r157:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=127] I190903 20:13:26.867808 161363 storage/replica_raft.go:291 [n3,s3,r157/1:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:26.868356 161363 storage/replica_command.go:1430 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r157:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=128] I190903 20:13:26.884974 161363 storage/replica_raft.go:291 [n3,s3,r157/1:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:26.885015 161363 storage/replica_raft.go:304 [n3,s3,r157/1:/Table/84/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:26.887808 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:26.888987 161363 storage/replica_command.go:979 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r157 failed: storage/replica_raft.go:305: [n3,s3,r157/1:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:26.889063 161363 storage/queue.go:1027 [n3,replicate,s3,r157/1:/Table/84/1/5/{6-7}] change replicas of r157 failed: storage/replica_raft.go:305: [n3,s3,r157/1:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:26.889732 161408 storage/replica_command.go:1430 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r158:/Table/84/{1/5/7-2} [(n3,s3):1, next=2, gen=127] I190903 20:13:26.900837 161408 storage/replica_raft.go:291 [n3,s3,r158/1:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:26.902238 161408 storage/store_snapshot.go:995 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] sending LEARNER snapshot db054bdd at applied index 19 I190903 20:13:26.902400 161408 storage/store_snapshot.go:1038 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:26.903526 161618 storage/replica_raftstorage.go:793 [n1,s1,r158/2:{-}] applying LEARNER snapshot [id=db054bdd index=19] I190903 20:13:26.904824 161618 storage/replica_raftstorage.go:814 [n1,s1,r158/2:/Table/84/{1/5/7-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=db054bdd index=19] I190903 20:13:26.905890 161408 storage/replica_command.go:1430 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r158:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=128] I190903 20:13:26.914260 161408 storage/replica_raft.go:291 [n3,s3,r158/1:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:26.914811 161408 storage/replica_command.go:1430 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r158:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=129] I190903 20:13:26.920230 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:26.934246 161408 storage/replica_raft.go:291 [n3,s3,r158/1:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:26.934282 161408 storage/replica_raft.go:304 [n3,s3,r158/1:/Table/84/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:26.935072 161408 storage/replica_command.go:979 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r158 failed: storage/replica_raft.go:305: [n3,s3,r158/1:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:26.935107 161408 storage/queue.go:1027 [n3,replicate,s3,r158/1:/Table/84/{1/5/7-2}] change replicas of r158 failed: storage/replica_raft.go:305: [n3,s3,r158/1:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:26.946181 160867 storage/queue.go:1027 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:26.965664 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:27.015494 162192 storage/replica_command.go:1430 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r154:/Table/84/1{-/3} [(n3,s3):1, (n2,s2):2, next=3, gen=125] I190903 20:13:27.039072 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:27.086990 162192 storage/replica_raft.go:291 [n2,s2,r154/2:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:27.096066 162392 storage/store.go:2593 [n3,replicaGC,s3,r154/1:/Table/84/1{-/3}] removing replica r154/1 I190903 20:13:27.096535 162392 storage/replica_destroy.go:152 [n3,replicaGC,s3,r154/1:/Table/84/1{-/3}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:27.234288 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) E190903 20:13:27.259222 162928 storage/queue.go:1027 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:27.264080 162899 storage/replica_command.go:1430 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r156:/Table/84/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:27.315904 162899 storage/replica_raft.go:291 [n1,s1,r156/2:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:27.324615 163064 storage/store.go:2593 [n3,replicaGC,s3,r156/1:/Table/84/1/{4-5/6}] removing replica r156/1 I190903 20:13:27.325094 163064 storage/replica_destroy.go:152 [n3,replicaGC,s3,r156/1:/Table/84/1/{4-5/6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:27.382547 163173 storage/replica_command.go:1430 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r157:/Table/84/1/5/{6-7} [(n3,s3):1, (n2,s2):2, next=3, gen=128] I190903 20:13:27.394700 163173 storage/replica_raft.go:291 [n2,s2,r157/2:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:27.453604 163308 storage/store.go:2593 [n3,replicaGC,s3,r157/1:/Table/84/1/5/{6-7}] removing replica r157/1 I190903 20:13:27.454027 163308 storage/replica_destroy.go:152 [n3,replicaGC,s3,r157/1:/Table/84/1/5/{6-7}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:27.509230 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:27.512580 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:27.516872 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) E190903 20:13:27.517884 163646 storage/queue.go:1027 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:26] I190903 20:13:27.518960 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:27.522491 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:27.575172 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=2 mtime=2019-09-03 20:13:27.573392952 +0000 UTC I190903 20:13:27.588894 163507 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=2:1567541912.245427488,0, refcount=0 I190903 20:13:27.590052 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 1}] I190903 20:13:27.660258 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=3 mtime=2019-09-03 20:13:27.659991505 +0000 UTC I190903 20:13:27.669024 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 2}] I190903 20:13:27.671186 164468 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=3:1567541912.661356291,0, refcount=0 I190903 20:13:27.706045 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=4 mtime=2019-09-03 20:13:27.700261979 +0000 UTC I190903 20:13:27.713908 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 84, info: {TableName:data.public."multi col range partitioning - MAXVALUE MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p3x VALUES FROM (3, 4) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7), PARTITION pxx VALUES FROM (5, 7) TO (maxvalue, maxvalue)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:27.715374 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:27.720538 164970 sql/lease.go:923 new lease: 84("multi col range partitioning - MAXVALUE MAXVALUE") ver=4:1567541912.662356291,0, refcount=0 I190903 20:13:27.722502 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 3}] I190903 20:13:27.763827 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:27.784248 165791 storage/replica_command.go:284 [n3,split,s3,r155/1:/Table/84/1/{3-4}] initiating a split of this range at key /Table/84/1/3/4 [r160] (zone config) I190903 20:13:27.799122 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:27.808531 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:27.810134 166160 storage/replica_command.go:1430 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r160:/Table/84/1/{3/4-4} [(n3,s3):1, next=2, gen=125] E190903 20:13:27.812219 165794 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 E190903 20:13:27.815404 166553 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:27.817293 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} E190903 20:13:27.820433 166762 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:27.823730 166160 storage/replica_raft.go:291 [n3,s3,r160/1:/Table/84/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:27.824442 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 84, info: {Target:PARTITION pxx OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:27.824552 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:27.824768 166160 storage/store_snapshot.go:995 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] sending LEARNER snapshot ac9aece1 at applied index 14 I190903 20:13:27.824898 166160 storage/store_snapshot.go:1038 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:27.825946 166766 storage/replica_raftstorage.go:793 [n2,s2,r160/2:{-}] applying LEARNER snapshot [id=ac9aece1 index=14] I190903 20:13:27.827202 166766 storage/replica_raftstorage.go:814 [n2,s2,r160/2:/Table/84/1/{3/4-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ac9aece1 index=14] I190903 20:13:27.827759 166160 storage/replica_command.go:1430 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r160:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=126] E190903 20:13:27.828171 167125 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):1 I190903 20:13:27.836624 166160 storage/replica_raft.go:291 [n3,s3,r160/1:/Table/84/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:27.837630 166160 storage/replica_command.go:1430 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r160:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=127] I190903 20:13:27.838798 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:27.847259 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:27.848243 166160 storage/replica_raft.go:291 [n3,s3,r160/1:/Table/84/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:27.848284 166160 storage/replica_raft.go:304 [n3,s3,r160/1:/Table/84/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:27.849628 166160 storage/replica_command.go:979 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r160 failed: storage/replica_raft.go:305: [n3,s3,r160/1:/Table/84/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:27.849702 166160 storage/queue.go:1027 [n3,replicate,s3,r160/1:/Table/84/1/{3/4-4}] change replicas of r160 failed: storage/replica_raft.go:305: [n3,s3,r160/1:/Table/84/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:27.850050 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.851207 167281 storage/replica_command.go:1430 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r155:/Table/84/1/3{-/4} [(n3,s3):1, next=2, gen=125] E190903 20:13:27.854076 167417 storage/queue.go:1027 [n2,replicate,s2,r160/2:/Table/84/1/{3/4-4}] no removable replicas from range that needs a removal: [1:0, 2*:20] I190903 20:13:27.858141 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:27.862286 167476 storage/replica_command.go:1430 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r156:/Table/84/1/{4-5/6} [(n1,s1):2, next=3, gen=128] I190903 20:13:27.862304 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:27.867594 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:27.870759 167476 storage/replica_raft.go:291 [n1,s1,r156/2:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:27.871269 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.872801 167281 storage/replica_raft.go:291 [n3,s3,r155/1:/Table/84/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:27.874665 167476 storage/store_snapshot.go:995 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] sending LEARNER snapshot 185592e8 at applied index 34 I190903 20:13:27.874849 167476 storage/store_snapshot.go:1038 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:27.876970 167281 storage/store_snapshot.go:995 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] sending LEARNER snapshot c80cab69 at applied index 24 I190903 20:13:27.877119 167281 storage/store_snapshot.go:1038 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 10, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:27.877814 167482 storage/replica_raftstorage.go:793 [n3,s3,r156/3:{-}] applying LEARNER snapshot [id=185592e8 index=34] I190903 20:13:27.878388 167411 storage/replica_raftstorage.go:793 [n1,s1,r155/2:{-}] applying LEARNER snapshot [id=c80cab69 index=24] I190903 20:13:27.879755 167411 storage/replica_raftstorage.go:814 [n1,s1,r155/2:/Table/84/1/3{-/4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=c80cab69 index=24] I190903 20:13:27.880810 167281 storage/replica_command.go:1430 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r155:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=126] I190903 20:13:27.882966 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.886907 167482 storage/replica_raftstorage.go:814 [n3,s3,r156/3:/Table/84/1/{4-5/6}] applied LEARNER snapshot [total=9ms ingestion=4@8ms id=185592e8 index=34] I190903 20:13:27.887869 167476 storage/replica_command.go:1430 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] change replicas (add [(n3,s3):3] remove []): existing descriptor r156:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=129] I190903 20:13:27.888350 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.892774 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:27.894115 167476 storage/replica_raft.go:291 [n1,s1,r156/2:/Table/84/1/{4-5/6}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 20:13:27.894580 167281 storage/replica_raft.go:291 [n3,s3,r155/1:/Table/84/1/3{-/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:27.894688 167476 storage/replica_command.go:1430 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r156:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=130] I190903 20:13:27.896538 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.896739 167281 storage/replica_command.go:1430 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r155:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:27.905373 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.910745 167476 storage/replica_raft.go:291 [n1,s1,r156/2:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 20:13:27.910800 167476 storage/replica_raft.go:304 [n1,s1,r156/2:/Table/84/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:27.911325 167281 storage/replica_raft.go:291 [n3,s3,r155/1:/Table/84/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:27.911414 167281 storage/replica_raft.go:304 [n3,s3,r155/1:/Table/84/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:27.912722 167281 storage/replica_command.go:979 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r155 failed: storage/replica_raft.go:305: [n3,s3,r155/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:27.912761 167281 storage/queue.go:1027 [n3,replicate,s3,r155/1:/Table/84/1/3{-/4}] change replicas of r155 failed: storage/replica_raft.go:305: [n3,s3,r155/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:27.912860 167476 storage/replica_command.go:979 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r156 failed: storage/replica_raft.go:305: [n1,s1,r156/2:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:27.912919 167476 storage/queue.go:1027 [n1,replicate,s1,r156/2:/Table/84/1/{4-5/6}] change replicas of r156 failed: storage/replica_raft.go:305: [n1,s1,r156/2:/Table/84/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:27.913219 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:27.915656 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.944422 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:27.948782 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:27.957174 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:27.960218 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:27.961011 167943 storage/replica_command.go:1430 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r158:/Table/84/{1/5/7-2} [(n3,s3):1, (n1,s1):2, next=3, gen=129] I190903 20:13:27.967221 167943 storage/replica_raft.go:291 [n1,s1,r158/2:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:27.974189 167943 storage/replica_command.go:1430 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r158:/Table/84/{1/5/7-2} [(n1,s1):2, next=3, gen=130] I190903 20:13:27.983940 167961 storage/store.go:2593 [n3,replicaGC,s3,r158/1:/Table/84/{1/5/7-2}] removing replica r158/1 I190903 20:13:27.984230 167961 storage/replica_destroy.go:152 [n3,replicaGC,s3,r158/1:/Table/84/{1/5/7-2}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:27.991071 167943 storage/replica_raft.go:291 [n1,s1,r158/2:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:27.992367 167943 storage/store_snapshot.go:995 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] sending LEARNER snapshot aa66c974 at applied index 33 I190903 20:13:27.992611 167943 storage/store_snapshot.go:1038 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:27.993627 168055 storage/replica_raftstorage.go:793 [n2,s2,r158/3:{-}] applying LEARNER snapshot [id=aa66c974 index=33] I190903 20:13:27.996409 168055 storage/replica_raftstorage.go:814 [n2,s2,r158/3:/Table/84/{1/5/7-2}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=aa66c974 index=33] I190903 20:13:27.997584 167943 storage/replica_command.go:1430 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] change replicas (add [(n2,s2):3] remove []): existing descriptor r158:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=131] I190903 20:13:28.003894 167943 storage/replica_raft.go:291 [n1,s1,r158/2:/Table/84/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:28.004827 167943 storage/replica_command.go:1430 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r158:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3, next=4, gen=132] I190903 20:13:28.025699 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:28.027096 167943 storage/replica_raft.go:291 [n1,s1,r158/2:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:28.027145 167943 storage/replica_raft.go:304 [n1,s1,r158/2:/Table/84/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:28.028458 167943 storage/replica_command.go:979 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r158 failed: storage/replica_raft.go:305: [n1,s1,r158/2:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:28.028491 167943 storage/queue.go:1027 [n1,replicate,s1,r158/2:/Table/84/{1/5/7-2}] change replicas of r158 failed: storage/replica_raft.go:305: [n1,s1,r158/2:/Table/84/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:28.034713 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:28.037789 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:28.044490 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:28.063162 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:28.070773 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:28.074490 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:28.085411 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:28.096346 168456 storage/replica_command.go:1430 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r154:/Table/84/1{-/3} [(n2,s2):2, next=3, gen=126] W190903 20:13:28.105398 168409 storage/replica_command.go:1917 [n3,s3,r155/1:/Table/84/1/3{-/4}] while transferring lease: unable to find store 2 in range r155:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:28.105837 168409 storage/replica_command.go:1430 [n3,s3,r155/1:/Table/84/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r155:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:28.119293 168409 storage/replica_raft.go:291 [n3,s3,r155/1:/Table/84/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:28.119345 168409 storage/replica_raft.go:304 [n3,s3,r155/1:/Table/84/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:28.120035 168456 storage/replica_raft.go:291 [n2,s2,r154/2:/Table/84/1{-/3}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 E190903 20:13:28.121032 168511 storage/queue.go:1027 [n3,replicate,s3,r156/3:/Table/84/1/{4-5/6}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 20:13:28.121561 168456 storage/store_snapshot.go:995 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] sending LEARNER snapshot 553810ef at applied index 34 I190903 20:13:28.121726 168456 storage/store_snapshot.go:1038 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:28.122808 167698 storage/replica_raftstorage.go:793 [n1,s1,r154/3:{-}] applying LEARNER snapshot [id=553810ef index=34] I190903 20:13:28.123866 168409 storage/replica_command.go:979 [n3,s3,r155/1:/Table/84/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r155 failed: storage/replica_raft.go:305: [n3,s3,r155/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) W190903 20:13:28.124560 168409 storage/replica_command.go:2033 [n3,s3,r155/1:/Table/84/1/3{-/4}] while removing target n3,s3: change replicas of r155 failed: storage/replica_raft.go:305: [n3,s3,r155/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:28.124839 168455 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] change replicas of r155 failed: storage/replica_raft.go:305: [n3,s3,r155/1:/Table/84/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:28.124923 167698 storage/replica_raftstorage.go:814 [n1,s1,r154/3:/Table/84/1{-/3}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=553810ef index=34] I190903 20:13:28.125373 168456 storage/replica_command.go:1430 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] change replicas (add [(n1,s1):3] remove []): existing descriptor r154:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=127] I190903 20:13:28.133475 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:28.135652 168456 storage/replica_raft.go:291 [n2,s2,r154/2:/Table/84/1{-/3}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:28.137165 168456 storage/replica_command.go:1430 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r154:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3, next=4, gen=128] I190903 20:13:28.150633 168456 storage/replica_raft.go:291 [n2,s2,r154/2:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:28.150678 168456 storage/replica_raft.go:304 [n2,s2,r154/2:/Table/84/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:28.151821 168456 storage/replica_command.go:979 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] could not promote [] to voter, rolling back: change replicas of r154 failed: storage/replica_raft.go:305: [n2,s2,r154/2:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:28.151857 168456 storage/queue.go:1027 [n2,replicate,s2,r154/2:/Table/84/1{-/3}] change replicas of r154 failed: storage/replica_raft.go:305: [n2,s2,r154/2:/Table/84/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:28.181915 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 20:13:28.217280 168900 storage/queue.go:1027 [n1,replicate,s1,r155/2:/Table/84/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 2*:33] I190903 20:13:28.255735 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:28.259856 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:28.264304 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:28.398090 169334 storage/replica_command.go:1430 [n2,replicate,s2,r160/2:/Table/84/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r160:/Table/84/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=127] I190903 20:13:28.404961 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:28.408400 169334 storage/replica_raft.go:291 [n2,s2,r160/2:/Table/84/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:28.410346 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:28.419134 169247 storage/store.go:2593 [n3,replicaGC,s3,r160/1:/Table/84/1/{3/4-4}] removing replica r160/1 I190903 20:13:28.419469 169247 storage/replica_destroy.go:152 [n3,replicaGC,s3,r160/1:/Table/84/1/{3/4-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:28.419497 169460 storage/replica_command.go:1430 [n3,replicate,s3,r156/3:/Table/84/1/{4-5/6}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r156:/Table/84/1/{4-5/6} [(n1,s1):2, (n3,s3):3, next=4, gen=130] I190903 20:13:28.429362 169460 storage/replica_raft.go:291 [n3,s3,r156/3:/Table/84/1/{4-5/6}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 20:13:28.439751 169425 storage/store.go:2593 [n1,replicaGC,s1,r156/2:/Table/84/1/{4-5/6}] removing replica r156/2 I190903 20:13:28.440155 169425 storage/replica_destroy.go:152 [n1,replicaGC,s1,r156/2:/Table/84/1/{4-5/6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:28.478449 169646 storage/queue.go:1027 [n2,replicate,s2,r158/3:/Table/84/{1/5/7-2}] no removable replicas from range that needs a removal: [2:0, 3*:40] I190903 20:13:28.612487 169879 storage/replica_command.go:1430 [n2,replicate,s2,r158/3:/Table/84/{1/5/7-2}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r158:/Table/84/{1/5/7-2} [(n1,s1):2, (n2,s2):3, next=4, gen=132] I190903 20:13:28.619755 169879 storage/replica_raft.go:291 [n2,s2,r158/3:/Table/84/{1/5/7-2}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:28.620977 169500 storage/replica_command.go:1430 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r157:/Table/84/1/5/{6-7} [(n2,s2):2, next=3, gen=129] I190903 20:13:28.629155 169500 storage/replica_raft.go:291 [n2,s2,r157/2:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:28.630620 169500 storage/store_snapshot.go:995 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] sending LEARNER snapshot 677c8a31 at applied index 37 I190903 20:13:28.630811 169500 storage/store_snapshot.go:1038 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:28.632744 170132 storage/replica_raftstorage.go:793 [n1,s1,r157/3:{-}] applying LEARNER snapshot [id=677c8a31 index=37] I190903 20:13:28.635740 170132 storage/replica_raftstorage.go:814 [n1,s1,r157/3:/Table/84/1/5/{6-7}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=677c8a31 index=37] I190903 20:13:28.642717 169500 storage/replica_command.go:1430 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] change replicas (add [(n1,s1):3] remove []): existing descriptor r157:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=130] I190903 20:13:28.656410 169500 storage/replica_raft.go:291 [n2,s2,r157/2:/Table/84/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:28.656970 169500 storage/replica_command.go:1430 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r157:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=131] I190903 20:13:28.670446 170237 storage/replica_command.go:1430 [n1,replicate,s1,r155/2:/Table/84/1/3{-/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r155:/Table/84/1/3{-/4} [(n3,s3):1, (n1,s1):2, next=3, gen=127] I190903 20:13:28.670576 169500 storage/replica_raft.go:291 [n2,s2,r157/2:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:28.670619 169500 storage/replica_raft.go:304 [n2,s2,r157/2:/Table/84/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:28.671428 169500 storage/replica_command.go:979 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r157 failed: storage/replica_raft.go:305: [n2,s2,r157/2:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:28.671470 169500 storage/queue.go:1027 [n2,replicate,s2,r157/2:/Table/84/1/5/{6-7}] change replicas of r157 failed: storage/replica_raft.go:305: [n2,s2,r157/2:/Table/84/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:28.675516 170058 storage/store.go:2593 [n1,replicaGC,s1,r158/2:/Table/84/{1/5/7-2}] removing replica r158/2 I190903 20:13:28.675921 170058 storage/replica_destroy.go:152 [n1,replicaGC,s1,r158/2:/Table/84/{1/5/7-2}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:28.676698 170237 storage/replica_raft.go:291 [n1,s1,r155/2:/Table/84/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:28.687365 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:28.691402 170134 storage/store.go:2593 [n3,replicaGC,s3,r155/1:/Table/84/1/3{-/4}] removing replica r155/1 I190903 20:13:28.691756 170134 storage/replica_destroy.go:152 [n3,replicaGC,s3,r155/1:/Table/84/1/3{-/4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:28.692839 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) E190903 20:13:28.955564 171034 storage/queue.go:1027 [n1,replicate,s1,r157/3:/Table/84/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:44] E190903 20:13:29.111566 171373 storage/queue.go:1027 [n1,replicate,s1,r154/3:/Table/84/1{-/3}] no removable replicas from range that needs a removal: [2:0, 3*:42] E190903 20:13:29.111617 171455 storage/queue.go:1027 [n2,merge,s2,r154/2:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):2 E190903 20:13:29.207701 171719 storage/queue.go:1027 [n1,merge,s1,r154/3:/Table/84/1{-/3}] none of the remaining targets [(n2,s2):?] are legal additions to (n1,s1):2 I190903 20:13:29.236621 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.240424 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:29.244345 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:29.247569 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.253950 171670 storage/replica_command.go:1430 [n1,replicate,s1,r154/3:/Table/84/1{-/3}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r154:/Table/84/1{-/3} [(n2,s2):2, (n1,s1):3, next=4, gen=128] I190903 20:13:29.255698 156438 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.259269 171670 storage/replica_raft.go:291 [n1,s1,r154/3:/Table/84/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 === RUN TestRepartitioning/multi_col_range_partitioning_-_MAXVALUE_MAXVALUE/multi_col_range_partitioning_-_MAXVALUE I190903 20:13:29.268519 171827 storage/store.go:2593 [n2,replicaGC,s2,r154/2:/Table/84/1{-/3}] removing replica r154/2 I190903 20:13:29.268939 171827 storage/replica_destroy.go:152 [n2,replicaGC,s2,r154/2:/Table/84/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:29.342050 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=5 mtime=2019-09-03 20:13:29.260863379 +0000 UTC I190903 20:13:29.353291 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 83, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE MAXVALUE"]} I190903 20:13:29.356321 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:29.358505 172350 storage/replica_command.go:609 [n1,merge,s1,r154/3:/Table/84/1{-/3}] initiating a merge of r155:/Table/84/1/3{-/4} [(n1,s1):2, next=3, gen=128] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:29.363090 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE MAXVALUE 84 4}] I190903 20:13:29.367138 192 storage/store.go:2593 [n1,s1,r154/3:/Table/84/1{-/3}] removing replica r155/2 I190903 20:13:29.420546 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=84 (multi col range partitioning - MAXVALUE MAXVALUE) version=6 mtime=2019-09-03 20:13:29.420337899 +0000 UTC I190903 20:13:29.463106 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 85, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:29.535057 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 86, info: {TableName:data.public."multi col range partitioning - MAXVALUE MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p3x VALUES FROM (3, 4) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7), PARTITION pxx VALUES FROM (5, 7) TO (maxvalue, maxvalue)) User:root} I190903 20:13:29.565834 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 87, info: {TableName:data.public."multi col range partitioning - MAXVALUE MAXVALUE" Statement:CREATE TABLE "multi col range partitioning - MAXVALUE MAXVALUE" (a INT8, b INT8, PRIMARY KEY (a, b)) PARTITION BY RANGE (a, b) (PARTITION p34 VALUES FROM (minvalue, minvalue) TO (3, 4), PARTITION p3x VALUES FROM (3, 4) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7), PARTITION pxx VALUES FROM (5, 7) TO (maxvalue, maxvalue)) User:root} I190903 20:13:29.571647 174053 storage/replica_command.go:284 [n3,split,s3,r159/1:/{Table/84/2-Max}] initiating a split of this range at key /Table/87 [r161] (zone config) I190903 20:13:29.603162 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p34 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:29.609120 174567 storage/replica_command.go:284 [n3,split,s3,r161/1:/{Table/87-Max}] initiating a split of this range at key /Table/87/1 [r162] (zone config) I190903 20:13:29.617537 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:29.632217 174971 storage/replica_command.go:284 [n3,split,s3,r162/1:/{Table/87/1-Max}] initiating a split of this range at key /Table/87/1/3/4 [r163] (zone config) I190903 20:13:29.643213 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:29.652958 174439 storage/replica_command.go:1430 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r162:/Table/87/1{-/3/4} [(n3,s3):1, next=2, gen=130] I190903 20:13:29.654190 175215 storage/replica_command.go:284 [n3,split,s3,r163/1:/{Table/87/1/3…-Max}] initiating a split of this range at key /Table/87/1/4 [r164] (zone config) I190903 20:13:29.660286 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:29.668768 174439 storage/replica_raft.go:291 [n3,s3,r162/1:/Table/87/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:29.672590 174439 storage/store_snapshot.go:995 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] sending LEARNER snapshot ae5ab9a2 at applied index 18 I190903 20:13:29.672762 174439 storage/store_snapshot.go:1038 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:29.674221 174978 storage/replica_raftstorage.go:793 [n1,s1,r162/2:{-}] applying LEARNER snapshot [id=ae5ab9a2 index=18] I190903 20:13:29.675354 175373 storage/replica_command.go:284 [n3,split,s3,r164/1:/{Table/87/1/4-Max}] initiating a split of this range at key /Table/87/1/5/6 [r165] (zone config) I190903 20:13:29.685083 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION pxx OF INDEX data.public."multi col range partitioning - MAXVALUE MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:29.686508 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.695480 174978 storage/replica_raftstorage.go:814 [n1,s1,r162/2:/Table/87/1{-/3/4}] applied LEARNER snapshot [total=21ms ingestion=4@20ms id=ae5ab9a2 index=18] I190903 20:13:29.697826 174439 storage/replica_command.go:1430 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] change replicas (add [(n1,s1):2] remove []): existing descriptor r162:/Table/87/1{-/3/4} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=131] I190903 20:13:29.698590 176229 storage/replica_command.go:284 [n3,split,s3,r165/1:/{Table/87/1/5…-Max}] initiating a split of this range at key /Table/87/1/5/7 [r166] (zone config) I190903 20:13:29.717320 174439 storage/replica_raft.go:291 [n3,s3,r162/1:/Table/87/1{-/3/4}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:29.717931 174439 storage/replica_command.go:1430 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r162:/Table/87/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=132] I190903 20:13:29.718253 176356 storage/replica_command.go:284 [n3,split,s3,r166/1:/{Table/87/1/5…-Max}] initiating a split of this range at key /Table/87/2 [r167] (zone config) I190903 20:13:29.729843 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.731025 176404 storage/replica_command.go:1430 [n1,replicate,s1,r157/3:/Table/84/1/5/{6-7}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r157:/Table/84/1/5/{6-7} [(n2,s2):2, (n1,s1):3, next=4, gen=131] I190903 20:13:29.758082 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.759400 174439 storage/replica_raft.go:291 [n3,s3,r162/1:/Table/87/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:29.759448 174439 storage/replica_raft.go:304 [n3,s3,r162/1:/Table/87/1{-/3/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:29.760249 174439 storage/replica_command.go:979 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] could not promote [] to voter, rolling back: change replicas of r162 failed: storage/replica_raft.go:305: [n3,s3,r162/1:/Table/87/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:29.760291 174439 storage/queue.go:1027 [n3,replicate,s3,r162/1:/Table/87/1{-/3/4}] change replicas of r162 failed: storage/replica_raft.go:305: [n3,s3,r162/1:/Table/87/1{-/3/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:29.760701 176082 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r163:/Table/87/1/{3/4-4} [(n3,s3):1, next=2, gen=131] I190903 20:13:29.767179 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.770280 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.774308 176404 storage/replica_raft.go:291 [n1,s1,r157/3:/Table/84/1/5/{6-7}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 I190903 20:13:29.778398 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.783432 176082 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/87/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:29.784571 176082 storage/store_snapshot.go:995 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] sending LEARNER snapshot bfa30b4f at applied index 18 I190903 20:13:29.784742 176082 storage/store_snapshot.go:1038 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:29.785993 176581 storage/replica_raftstorage.go:793 [n2,s2,r163/2:{-}] applying LEARNER snapshot [id=bfa30b4f index=18] I190903 20:13:29.787333 176581 storage/replica_raftstorage.go:814 [n2,s2,r163/2:/Table/87/1/{3/4-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=bfa30b4f index=18] I190903 20:13:29.787944 176082 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r163:/Table/87/1/{3/4-4} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=132] I190903 20:13:29.792103 176494 storage/store.go:2593 [n2,replicaGC,s2,r157/2:/Table/84/1/5/{6-7}] removing replica r157/2 I190903 20:13:29.792618 176494 storage/replica_destroy.go:152 [n2,replicaGC,s2,r157/2:/Table/84/1/5/{6-7}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:29.794362 176082 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/87/1/{3/4-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:29.794953 176082 storage/replica_command.go:1430 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r163:/Table/87/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=133] I190903 20:13:29.798525 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:29.805792 176082 storage/replica_raft.go:291 [n3,s3,r163/1:/Table/87/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:29.822070 176082 storage/replica_raft.go:304 [n3,s3,r163/1:/Table/87/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:29.812787 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.823129 176082 storage/replica_command.go:979 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r163 failed: storage/replica_raft.go:305: [n3,s3,r163/1:/Table/87/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:29.823162 176082 storage/queue.go:1027 [n3,replicate,s3,r163/1:/Table/87/1/{3/4-4}] change replicas of r163 failed: storage/replica_raft.go:305: [n3,s3,r163/1:/Table/87/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:29.824015 176419 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r165:/Table/87/1/5/{6-7} [(n3,s3):1, next=2, gen=133] I190903 20:13:29.830616 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.831845 176419 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/87/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:29.833082 176419 storage/store_snapshot.go:995 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] sending LEARNER snapshot a5fe999b at applied index 17 I190903 20:13:29.833331 176419 storage/store_snapshot.go:1038 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:29.834416 176787 storage/replica_raftstorage.go:793 [n1,s1,r165/2:{-}] applying LEARNER snapshot [id=a5fe999b index=17] I190903 20:13:29.839255 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:29.842561 176787 storage/replica_raftstorage.go:814 [n1,s1,r165/2:/Table/87/1/5/{6-7}] applied LEARNER snapshot [total=8ms ingestion=4@1ms id=a5fe999b index=17] I190903 20:13:29.843134 176419 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] change replicas (add [(n1,s1):2] remove []): existing descriptor r165:/Table/87/1/5/{6-7} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=134] I190903 20:13:29.852417 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.854149 176419 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/87/1/5/{6-7}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:29.854929 176419 storage/replica_command.go:1430 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r165:/Table/87/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=135] I190903 20:13:29.859173 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.877975 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.886136 176419 storage/replica_raft.go:291 [n3,s3,r165/1:/Table/87/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:29.886201 176419 storage/replica_raft.go:304 [n3,s3,r165/1:/Table/87/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:29.887296 176419 storage/replica_command.go:979 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r165 failed: storage/replica_raft.go:305: [n3,s3,r165/1:/Table/87/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:29.888812 176419 storage/queue.go:1027 [n3,replicate,s3,r165/1:/Table/87/1/5/{6-7}] change replicas of r165 failed: storage/replica_raft.go:305: [n3,s3,r165/1:/Table/87/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:29.889398 176956 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, next=2, gen=134] I190903 20:13:29.917660 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:29.921682 176956 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:29.923107 176956 storage/store_snapshot.go:995 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] sending LEARNER snapshot b014eeed at applied index 18 I190903 20:13:29.923281 176956 storage/store_snapshot.go:1038 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:29.925885 177065 storage/replica_raftstorage.go:793 [n2,s2,r166/2:{-}] applying LEARNER snapshot [id=b014eeed index=18] I190903 20:13:29.932179 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:29.937672 177065 storage/replica_raftstorage.go:814 [n2,s2,r166/2:/Table/87/{1/5/7-2}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=b014eeed index=18] I190903 20:13:29.944180 176956 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [(n2,s2):2] remove []): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=135] I190903 20:13:29.954564 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.960826 176956 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:29.961395 176956 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n2,s2):2, next=3, gen=136] I190903 20:13:29.965532 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.970951 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:29.972628 176956 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 E190903 20:13:29.972676 176956 storage/replica_raft.go:304 [n3,s3,r166/1:/Table/87/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:29.974798 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.978075 176956 storage/replica_command.go:979 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/87/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:29.978113 176956 storage/queue.go:1027 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/87/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:29.983340 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:29.987844 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:29.994630 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:29.999060 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.006797 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:30.016626 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.024317 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.033123 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.047379 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:30.070482 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 20:13:30.074527 177619 storage/queue.go:1027 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:30.108939 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 20:13:30.126309 177834 storage/queue.go:1027 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:30.180296 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) E190903 20:13:30.188364 177635 storage/queue.go:1027 [n1,replicate,s1,r162/2:/Table/87/1{-/3/4}] no removable replicas from range that needs a removal: [1:0, 2*:25] E190903 20:13:30.283840 177948 storage/queue.go:1027 [n2,replicate,s2,r166/2:/Table/87/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:30.298516 178288 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r163:/Table/87/1/{3/4-4} [(n3,s3):1, (n2,s2):2, next=3, gen=133] I190903 20:13:30.310090 178288 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/87/1/{3/4-4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:30.318629 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.340100 178462 storage/replica_command.go:1430 [n1,replicate,s1,r162/2:/Table/87/1{-/3/4}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r162:/Table/87/1{-/3/4} [(n3,s3):1, (n1,s1):2, next=3, gen=132] I190903 20:13:30.349600 178462 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3/4}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:30.372111 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:30.376464 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) < (3, 4) I190903 20:13:30.378328 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE (a, b) >= (3, 4) AND a < 4 I190903 20:13:30.381635 178323 storage/store.go:2593 [n3,replicaGC,s3,r163/1:/Table/87/1/{3/4-4}] removing replica r163/1 I190903 20:13:30.382213 178323 storage/replica_destroy.go:152 [n3,replicaGC,s3,r163/1:/Table/87/1/{3/4-4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:30.383404 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.395382 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=87 (multi col range partitioning - MAXVALUE) version=2 mtime=2019-09-03 20:13:30.393320239 +0000 UTC I190903 20:13:30.396856 178491 storage/store.go:2593 [n3,replicaGC,s3,r162/1:/Table/87/1{-/3/4}] removing replica r162/1 I190903 20:13:30.397228 178491 storage/replica_destroy.go:152 [n3,replicaGC,s3,r162/1:/Table/87/1{-/3/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:30.409025 178641 sql/lease.go:923 new lease: 87("multi col range partitioning - MAXVALUE") ver=2:1567541924.884212041,0, refcount=0 I190903 20:13:30.411807 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:30.416807 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 87 1}] I190903 20:13:30.466466 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=87 (multi col range partitioning - MAXVALUE) version=3 mtime=2019-09-03 20:13:30.466266148 +0000 UTC I190903 20:13:30.478124 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 87 2}] I190903 20:13:30.480077 178946 sql/lease.go:923 new lease: 87("multi col range partitioning - MAXVALUE") ver=3:1567541924.885212041,0, refcount=0 I190903 20:13:30.506389 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=87 (multi col range partitioning - MAXVALUE) version=4 mtime=2019-09-03 20:13:30.502878005 +0000 UTC I190903 20:13:30.522412 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 87, info: {TableName:data.public."multi col range partitioning - MAXVALUE" Statement:ALTER TABLE "multi col range partitioning - MAXVALUE" PARTITION BY RANGE (a, b) (PARTITION p3n VALUES FROM (minvalue, minvalue) TO (3, minvalue), PARTITION p3x VALUES FROM (3, minvalue) TO (3, maxvalue), PARTITION p56 VALUES FROM (3, maxvalue) TO (5, 6), PARTITION p57 VALUES FROM (5, 6) TO (5, 7)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:30.532433 179891 sql/lease.go:923 new lease: 87("multi col range partitioning - MAXVALUE") ver=4:1567541924.886212041,0, refcount=0 I190903 20:13:30.534464 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 87 3}] I190903 20:13:30.534962 179950 storage/replica_command.go:1430 [n3,s3,r167/1:/{Table/87/2-Max}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r167:/{Table/87/2-Max} [(n3,s3):1, next=2, gen=134] I190903 20:13:30.553096 179950 storage/replica_raft.go:291 [n3,s3,r167/1:/{Table/87/2-Max}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n3,s3):1 (n2,s2):2LEARNER] next=3 I190903 20:13:30.554550 179950 storage/store_snapshot.go:995 [n3,s3,r167/1:/{Table/87/2-Max}] sending LEARNER snapshot 30b32f63 at applied index 15 I190903 20:13:30.554701 179950 storage/store_snapshot.go:1038 [n3,s3,r167/1:/{Table/87/2-Max}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.555909 180278 storage/replica_raftstorage.go:793 [n2,s2,r167/2:{-}] applying LEARNER snapshot [id=30b32f63 index=15] I190903 20:13:30.557150 180278 storage/replica_raftstorage.go:814 [n2,s2,r167/2:/{Table/87/2-Max}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=30b32f63 index=15] I190903 20:13:30.558095 179950 storage/replica_command.go:1430 [n3,s3,r167/1:/{Table/87/2-Max}] change replicas (add [(n2,s2):2] remove []): existing descriptor r167:/{Table/87/2-Max} [(n3,s3):1, (n2,s2):2LEARNER, next=3, gen=135] I190903 20:13:30.575352 179950 storage/replica_raft.go:291 [n3,s3,r167/1:/{Table/87/2-Max}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n3,s3):1 (n2,s2):2] next=3 I190903 20:13:30.579119 180331 storage/replica_command.go:1430 [n1,s1,r162/2:/Table/87/1{-/3/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r162:/Table/87/1{-/3/4} [(n1,s1):2, next=3, gen=133] E190903 20:13:30.584261 180406 storage/queue.go:1027 [n2,replicate,s2,r167/2:/{Table/87/2-Max}] no removable replicas from range that needs a removal: [1:0, 2*:19] I190903 20:13:30.585975 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:30.587587 180331 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:30.588953 180616 storage/replica_command.go:284 [n1,split,s1,r162/2:/Table/87/1{-/3/4}] initiating a split of this range at key /Table/87/1/3 [r102] (zone config) I190903 20:13:30.599912 180616 storage/split_queue.go:149 [n1,split,s1,r162/2:/Table/87/1{-/3/4}] split saw concurrent descriptor modification; maybe retrying I190903 20:13:30.600108 180126 storage/replica_command.go:284 [n1,split,s1,r162/2:/Table/87/1{-/3/4}] initiating a split of this range at key /Table/87/1/3 [r103] (zone config); r162/3 is being probed (may or may not need a Raft snapshot) I190903 20:13:30.606568 180331 storage/store_snapshot.go:995 [n1,s1,r162/2:/Table/87/1{-/3/4}] sending LEARNER snapshot 4f79db3b at applied index 31 I190903 20:13:30.609627 180750 storage/raft_snapshot_queue.go:120 [n1,raftsnapshot,s1,r162/2:/Table/87/1{-/3/4}] skipping snapshot; replica is likely a learner in the process of being added: (n3,s3):3LEARNER I190903 20:13:30.610694 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p3n OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:30.610880 180331 storage/store_snapshot.go:1038 [n1,s1,r162/2:/Table/87/1{-/3/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:30.611558 180804 storage/replica_raftstorage.go:793 [n3,s3,r162/3:{-}] applying LEARNER snapshot [id=4f79db3b index=31] I190903 20:13:30.627882 180778 storage/replica_command.go:1430 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] change replicas (add [] remove [(n3,s3):3LEARNER]): existing descriptor r162:/Table/87/1{-/3} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=135] I190903 20:13:30.678122 180804 storage/replica_raftstorage.go:814 [n3,s3,r162/3:/Table/87/1{-/3/4}] applied LEARNER snapshot [total=66ms ingestion=4@65ms id=4f79db3b index=31] I190903 20:13:30.710718 179938 storage/replica_command.go:609 [n2,merge,s2,r166/2:/Table/87/{1/5/7-2}] initiating a merge of r167:/{Table/87/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=136] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:30.711147 180778 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3}] proposing REMOVE_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2] next=4 I190903 20:13:30.711458 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p3x OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:30.711807 180778 storage/replica_command.go:1430 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r162:/Table/87/1{-/3} [(n1,s1):2, next=4, gen=136] I190903 20:13:30.717134 181397 storage/replica_command.go:1430 [n1,s1,r103/1:/Table/87/1/3{-/4}] change replicas (add [] remove [(n3,s3):2LEARNER]): existing descriptor r103:/Table/87/1/3{-/4} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=135] I190903 20:13:30.722857 181397 storage/replica_raft.go:291 [n1,s1,r103/1:/Table/87/1/3{-/4}] proposing REMOVE_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1] next=3 I190903 20:13:30.723164 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p56 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n1]' User:root} E190903 20:13:30.723794 181397 storage/queue.go:1027 [n1,merge,s1,r103/1:/Table/87/1/3{-/4}] none of the remaining targets [(n1,s1):?] are legal additions to (n2,s2):2 E190903 20:13:30.727081 181733 storage/queue.go:1027 [n1,merge,s1,r103/1:/Table/87/1/3{-/4}] none of the remaining targets [(n1,s1):?] are legal additions to (n2,s2):2 I190903 20:13:30.728033 1695 storage/store.go:2593 [n2,s2,r166/2:/Table/87/{1/5/7-2}] removing replica r167/2 I190903 20:13:30.728501 2286 storage/store.go:2593 [n3,s3,r166/1:/Table/87/{1/5/7-2}] removing replica r167/1 I190903 20:13:30.738423 180331 storage/replica_command.go:979 [n1,s1,r162/2:/Table/87/1{-/3}] could not promote [n3,s3] to voter, rolling back: change replicas of r162 failed: descriptor changed: [expected] r162:/Table/87/1{-/3/4} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=134] != [actual] r162:/Table/87/1{-/3} [(n1,s1):2, next=4, gen=136] E190903 20:13:30.740376 179956 storage/queue.go:1027 [n3,merge,s3,r161/1:/Table/87{-/1}] while adding target n3,s3: change replicas of r162 failed: descriptor changed: [expected] r162:/Table/87/1{-/3/4} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=134] != [actual] r162:/Table/87/1{-/3} [(n1,s1):2, next=4, gen=136] I190903 20:13:30.740583 298 server/status/runtime.go:498 [n1] runtime stats: 256 MiB RSS, 984 goroutines, 63 MiB/28 MiB/139 MiB GO alloc/idle/total, 54 MiB/106 MiB CGO alloc/total, 7916.4 CGO/sec, 79.4/9.1 %(u/s)time, 0.7 %gc (12x), 96 MiB/96 MiB (r/w)net I190903 20:13:30.741360 180808 storage/store.go:2593 [n3,replicaGC,s3,r162/3:/Table/87/1{-/3}] removing replica r162/3 I190903 20:13:30.741642 180808 storage/replica_destroy.go:152 [n3,replicaGC,s3,r162/3:/Table/87/1{-/3}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:30.745821 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 87, info: {Target:PARTITION p57 OF INDEX data.public."multi col range partitioning - MAXVALUE"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:30.745988 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.746888 182156 storage/store.go:2593 [n3,replicaGC,s3,r103/2:/Table/87/1/3{-/4}] removing replica r103/2 I190903 20:13:30.747146 182156 storage/replica_destroy.go:152 [n3,replicaGC,s3,r103/2:/Table/87/1/3{-/4}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:30.750517 182249 storage/queue.go:1027 [n1,merge,s1,r103/1:/Table/87/1/3{-/4}] none of the remaining targets [(n1,s1):?] are legal additions to (n2,s2):2 I190903 20:13:30.750701 182565 storage/replica_command.go:284 [n2,split,s2,r166/2:/{Table/87/1/5…-Max}] initiating a split of this range at key /Table/87/2 [r69] (zone config) I190903 20:13:30.758715 180778 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n1,s1):2 (n2,s2):4LEARNER] next=5 I190903 20:13:30.760027 180778 storage/store_snapshot.go:995 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] sending LEARNER snapshot 5f5d24fb at applied index 39 I190903 20:13:30.760313 180778 storage/store_snapshot.go:1038 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 18, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.760374 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.763174 181680 storage/replica_raftstorage.go:793 [n2,s2,r162/4:{-}] applying LEARNER snapshot [id=5f5d24fb index=39] I190903 20:13:30.764447 181680 storage/replica_raftstorage.go:814 [n2,s2,r162/4:/Table/87/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=5f5d24fb index=39] I190903 20:13:30.764578 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:30.765193 180778 storage/replica_command.go:1430 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] change replicas (add [(n2,s2):4] remove []): existing descriptor r162:/Table/87/1{-/3} [(n1,s1):2, (n2,s2):4LEARNER, next=5, gen=137] I190903 20:13:30.771630 180778 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n1,s1):2 (n2,s2):4] next=5 I190903 20:13:30.775960 182561 storage/replica_command.go:1430 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r164:/Table/87/1/{4-5/6} [(n3,s3):1, next=2, gen=132] I190903 20:13:30.776031 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:30.778837 180778 storage/replica_command.go:1430 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r162:/Table/87/1{-/3} [(n1,s1):2, (n2,s2):4, next=5, gen=138] I190903 20:13:30.796270 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.796747 182561 storage/replica_raft.go:291 [n3,s3,r164/1:/Table/87/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n3,s3):1 (n1,s1):2LEARNER] next=3 I190903 20:13:30.799055 182561 storage/store_snapshot.go:995 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] sending LEARNER snapshot bffc57eb at applied index 19 I190903 20:13:30.799243 182561 storage/store_snapshot.go:1038 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 8, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.800453 182179 storage/replica_raftstorage.go:793 [n1,s1,r164/2:{-}] applying LEARNER snapshot [id=bffc57eb index=19] I190903 20:13:30.810583 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:30.811730 180778 storage/replica_raft.go:291 [n1,s1,r162/2:/Table/87/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):4] next=5 E190903 20:13:30.811781 180778 storage/replica_raft.go:304 [n1,s1,r162/2:/Table/87/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:13:30.815426 182179 storage/replica_raftstorage.go:814 [n1,s1,r164/2:/Table/87/1/{4-5/6}] applied LEARNER snapshot [total=15ms ingestion=4@1ms id=bffc57eb index=19] I190903 20:13:30.816581 180778 storage/replica_command.go:979 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] could not promote [] to voter, rolling back: change replicas of r162 failed: storage/replica_raft.go:305: [n1,s1,r162/2:/Table/87/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 20:13:30.818899 180778 storage/queue.go:1027 [n1,replicate,s1,r162/2:/Table/87/1{-/3}] change replicas of r162 failed: storage/replica_raft.go:305: [n1,s1,r162/2:/Table/87/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:13:30.816989 182561 storage/replica_command.go:1430 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] change replicas (add [(n1,s1):2] remove []): existing descriptor r164:/Table/87/1/{4-5/6} [(n3,s3):1, (n1,s1):2LEARNER, next=3, gen=133] I190903 20:13:30.817841 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:30.821428 182731 storage/replica_command.go:1430 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r103:/Table/87/1/3{-/4} [(n1,s1):1, next=3, gen=136] I190903 20:13:30.826651 182561 storage/replica_raft.go:291 [n3,s3,r164/1:/Table/87/1/{4-5/6}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n3,s3):1 (n1,s1):2] next=3 I190903 20:13:30.827213 182561 storage/replica_command.go:1430 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r164:/Table/87/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=134] I190903 20:13:30.864270 182731 storage/replica_raft.go:291 [n1,s1,r103/1:/Table/87/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 20:13:30.865924 182731 storage/store_snapshot.go:995 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] sending LEARNER snapshot 5513aedd at applied index 16 I190903 20:13:30.866306 182731 storage/store_snapshot.go:1038 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.866709 182561 storage/replica_raft.go:291 [n3,s3,r164/1:/Table/87/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 E190903 20:13:30.866737 182561 storage/replica_raft.go:304 [n3,s3,r164/1:/Table/87/1/{4-5/6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:30.867811 182561 storage/replica_command.go:979 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] could not promote [] to voter, rolling back: change replicas of r164 failed: storage/replica_raft.go:305: [n3,s3,r164/1:/Table/87/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:30.867842 182561 storage/queue.go:1027 [n3,replicate,s3,r164/1:/Table/87/1/{4-5/6}] change replicas of r164 failed: storage/replica_raft.go:305: [n3,s3,r164/1:/Table/87/1/{4-5/6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:30.873227 182954 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n2,s2):2, next=3, gen=138] I190903 20:13:30.880273 183015 storage/replica_raftstorage.go:793 [n3,s3,r103/3:{-}] applying LEARNER snapshot [id=5513aedd index=16] I190903 20:13:30.887451 183015 storage/replica_raftstorage.go:814 [n3,s3,r103/3:/Table/87/1/3{-/4}] applied LEARNER snapshot [total=7ms ingestion=4@6ms id=5513aedd index=16] I190903 20:13:30.888126 182731 storage/replica_command.go:1430 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r103:/Table/87/1/3{-/4} [(n1,s1):1, (n3,s3):3LEARNER, next=4, gen=137] I190903 20:13:30.899904 182731 storage/replica_raft.go:291 [n1,s1,r103/1:/Table/87/1/3{-/4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 20:13:30.902314 182954 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):1] next=3 I190903 20:13:30.904370 182731 storage/replica_command.go:1430 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r103:/Table/87/1/3{-/4} [(n1,s1):1, (n3,s3):3, next=4, gen=138] I190903 20:13:30.907890 182954 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, next=3, gen=139] I190903 20:13:30.915621 183204 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r163:/Table/87/1/{3/4-4} [(n2,s2):2, next=3, gen=134] I190903 20:13:30.919922 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.923492 182954 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):1 (n1,s1):3LEARNER] next=4 I190903 20:13:30.926006 182731 storage/replica_raft.go:291 [n1,s1,r103/1:/Table/87/1/3{-/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 E190903 20:13:30.926040 182731 storage/replica_raft.go:304 [n1,s1,r103/1:/Table/87/1/3{-/4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:30.926663 182954 storage/store_snapshot.go:995 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] sending LEARNER snapshot 962d217f at applied index 40 I190903 20:13:30.926877 182954 storage/store_snapshot.go:1038 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.927097 183047 storage/store.go:2593 [n2,replicaGC,s2,r166/2:/Table/87/{1/5/7-2}] removing replica r166/2 I190903 20:13:30.927320 182731 storage/replica_command.go:979 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] could not promote [] to voter, rolling back: change replicas of r103 failed: storage/replica_raft.go:305: [n1,s1,r103/1:/Table/87/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:30.927384 182731 storage/queue.go:1027 [n1,replicate,s1,r103/1:/Table/87/1/3{-/4}] change replicas of r103 failed: storage/replica_raft.go:305: [n1,s1,r103/1:/Table/87/1/3{-/4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:30.927523 183047 storage/replica_destroy.go:152 [n2,replicaGC,s2,r166/2:/Table/87/{1/5/7-2}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:30.928089 183252 storage/replica_raftstorage.go:793 [n1,s1,r166/3:{-}] applying LEARNER snapshot [id=962d217f index=40] I190903 20:13:30.929403 183252 storage/replica_raftstorage.go:814 [n1,s1,r166/3:/Table/87/{1/5/7-2}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=962d217f index=40] I190903 20:13:30.929946 182954 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [(n1,s1):3] remove []): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n1,s1):3LEARNER, next=4, gen=140] I190903 20:13:30.930159 183204 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/87/1/{3/4-4}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n2,s2):2 (n3,s3):3LEARNER] next=4 I190903 20:13:30.931325 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:30.932653 183204 storage/store_snapshot.go:995 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] sending LEARNER snapshot f04cad21 at applied index 36 I190903 20:13:30.932857 183204 storage/store_snapshot.go:1038 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:30.934098 183285 storage/replica_raftstorage.go:793 [n3,s3,r163/3:{-}] applying LEARNER snapshot [id=f04cad21 index=36] I190903 20:13:30.935867 183285 storage/replica_raftstorage.go:814 [n3,s3,r163/3:/Table/87/1/{3/4-4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=f04cad21 index=36] I190903 20:13:30.936369 182954 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):1 (n1,s1):3] next=4 I190903 20:13:30.936835 183204 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] change replicas (add [(n3,s3):3] remove []): existing descriptor r163:/Table/87/1/{3/4-4} [(n2,s2):2, (n3,s3):3LEARNER, next=4, gen=135] I190903 20:13:30.937062 182954 storage/replica_command.go:1430 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n1,s1):3, next=4, gen=141] I190903 20:13:30.938118 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:30.940064 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:30.945110 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:30.948216 183204 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/87/1/{3/4-4}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n2,s2):2 (n3,s3):3] next=4 I190903 20:13:30.948767 183204 storage/replica_command.go:1430 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r163:/Table/87/1/{3/4-4} [(n2,s2):2, (n3,s3):3, next=4, gen=136] I190903 20:13:30.951234 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:30.953748 182954 storage/replica_raft.go:291 [n3,s3,r166/1:/Table/87/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 E190903 20:13:30.953800 182954 storage/replica_raft.go:304 [n3,s3,r166/1:/Table/87/{1/5/7-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:30.966367 182954 storage/replica_command.go:979 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] could not promote [] to voter, rolling back: change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/87/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:30.966421 182954 storage/queue.go:1027 [n3,replicate,s3,r166/1:/Table/87/{1/5/7-2}] change replicas of r166 failed: storage/replica_raft.go:305: [n3,s3,r166/1:/Table/87/{1/5/7-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:30.969728 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 20:13:30.979138 183492 storage/queue.go:1027 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] no removable replicas from range that needs a removal: [1:0, 2*:32] I190903 20:13:30.981444 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:30.982824 183204 storage/replica_raft.go:291 [n2,s2,r163/2:/Table/87/1/{3/4-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 E190903 20:13:30.982889 183204 storage/replica_raft.go:304 [n2,s2,r163/2:/Table/87/1/{3/4-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:30.987591 183204 storage/replica_command.go:979 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] could not promote [] to voter, rolling back: change replicas of r163 failed: storage/replica_raft.go:305: [n2,s2,r163/2:/Table/87/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:30.988046 183204 storage/queue.go:1027 [n2,replicate,s2,r163/2:/Table/87/1/{3/4-4}] change replicas of r163 failed: storage/replica_raft.go:305: [n2,s2,r163/2:/Table/87/1/{3/4-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:30.988595 183399 storage/replica_command.go:1430 [n2,replicate,s2,r69/2:/{Table/87/2-Max}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r69:/{Table/87/2-Max} [(n3,s3):1, (n2,s2):2, next=3, gen=138] I190903 20:13:30.988821 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:30.995277 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.008241 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.009545 183399 storage/replica_raft.go:291 [n2,s2,r69/2:/{Table/87/2-Max}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n2,s2):2] next=3 I190903 20:13:31.011344 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.021417 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.022004 183406 storage/store.go:2593 [n3,replicaGC,s3,r69/1:/{Table/87/2-Max}] removing replica r69/1 I190903 20:13:31.022305 183406 storage/replica_destroy.go:152 [n3,replicaGC,s3,r69/1:/{Table/87/2-Max}] removed 7 (0+7) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:31.025589 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:31.031933 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.039283 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.046266 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:31.061828 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:31.082802 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 E190903 20:13:31.086905 183827 storage/queue.go:1027 [n1,merge,s1,r103/1:/Table/87/1/3{-/4}] none of the remaining targets [(n1,s1):?] are legal additions to (n2,s2):2,(n3,s3):3 E190903 20:13:31.087799 183642 storage/queue.go:1027 [n3,replicate,s3,r103/3:/Table/87/1/3{-/4}] no removable replicas from range that needs a removal: [1:0, 3*:23] I190903 20:13:31.119034 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) E190903 20:13:31.129470 183901 storage/queue.go:1027 [n3,replicate,s3,r163/3:/Table/87/1/{3/4-4}] no removable replicas from range that needs a removal: [2:0, 3*:43] E190903 20:13:31.129552 183993 storage/queue.go:1027 [n2,replicate,s2,r162/4:/Table/87/1{-/3}] no removable replicas from range that needs a removal: [2:0, 4*:46] I190903 20:13:31.193717 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:31.330442 1776 server/status/runtime.go:498 [n2] runtime stats: 266 MiB RSS, 978 goroutines, 61 MiB/28 MiB/139 MiB GO alloc/idle/total, 53 MiB/108 MiB CGO alloc/total, 8533.1 CGO/sec, 84.9/10.0 %(u/s)time, 0.8 %gc (13x), 96 MiB/96 MiB (r/w)net I190903 20:13:31.331607 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:31.357722 184575 storage/replica_command.go:1430 [n2,replicate,s2,r162/4:/Table/87/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r162:/Table/87/1{-/3} [(n1,s1):2, (n2,s2):4, next=5, gen=138] I190903 20:13:31.365523 184575 storage/replica_raft.go:291 [n2,s2,r162/4:/Table/87/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):4] next=5 I190903 20:13:31.374450 184466 storage/replica_command.go:1430 [n3,replicate,s3,r163/3:/Table/87/1/{3/4-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r163:/Table/87/1/{3/4-4} [(n2,s2):2, (n3,s3):3, next=4, gen=136] I190903 20:13:31.374627 184579 storage/store.go:2593 [n1,replicaGC,s1,r162/2:/Table/87/1{-/3}] removing replica r162/2 I190903 20:13:31.375070 184579 storage/replica_destroy.go:152 [n1,replicaGC,s1,r162/2:/Table/87/1{-/3}] removed 13 (0+13) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:31.383499 184466 storage/replica_raft.go:291 [n3,s3,r163/3:/Table/87/1/{3/4-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n3,s3):3] next=4 I190903 20:13:31.392884 184649 storage/store.go:2593 [n2,replicaGC,s2,r163/2:/Table/87/1/{3/4-4}] removing replica r163/2 I190903 20:13:31.393322 184649 storage/replica_destroy.go:152 [n2,replicaGC,s2,r163/2:/Table/87/1/{3/4-4}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:31.453557 184931 storage/queue.go:1027 [n1,replicate,s1,r166/3:/Table/87/{1/5/7-2}] no removable replicas from range that needs a removal: [1:0, 3*:47] I190903 20:13:31.455291 2354 server/status/runtime.go:498 [n3] runtime stats: 268 MiB RSS, 972 goroutines, 71 MiB/20 MiB/139 MiB GO alloc/idle/total, 55 MiB/110 MiB CGO alloc/total, 8569.1 CGO/sec, 84.0/9.9 %(u/s)time, 0.8 %gc (12x), 96 MiB/96 MiB (r/w)net I190903 20:13:31.464096 184992 storage/replica_command.go:1430 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r165:/Table/87/1/5/{6-7} [(n3,s3):1, (n1,s1):2, next=3, gen=135] I190903 20:13:31.469712 184992 storage/replica_raft.go:291 [n1,s1,r165/2:/Table/87/1/5/{6-7}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:31.471375 184992 storage/replica_command.go:1430 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r165:/Table/87/1/5/{6-7} [(n1,s1):2, next=3, gen=136] I190903 20:13:31.478549 185002 storage/store.go:2593 [n3,replicaGC,s3,r165/1:/Table/87/1/5/{6-7}] removing replica r165/1 I190903 20:13:31.478904 185002 storage/replica_destroy.go:152 [n3,replicaGC,s3,r165/1:/Table/87/1/5/{6-7}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:31.481850 184992 storage/replica_raft.go:291 [n1,s1,r165/2:/Table/87/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:31.483017 184992 storage/store_snapshot.go:995 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] sending LEARNER snapshot 56c110a2 at applied index 37 I190903 20:13:31.483215 184992 storage/store_snapshot.go:1038 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:31.484784 185064 storage/replica_raftstorage.go:793 [n2,s2,r165/3:{-}] applying LEARNER snapshot [id=56c110a2 index=37] I190903 20:13:31.486470 185064 storage/replica_raftstorage.go:814 [n2,s2,r165/3:/Table/87/1/5/{6-7}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=56c110a2 index=37] I190903 20:13:31.487016 184992 storage/replica_command.go:1430 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] change replicas (add [(n2,s2):3] remove []): existing descriptor r165:/Table/87/1/5/{6-7} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=137] E190903 20:13:31.493403 185055 storage/queue.go:1027 [n3,merge,s3,r103/3:/Table/87/1/3{-/4}] none of the remaining targets [(n1,s1):?] are legal additions to (n3,s3):3 I190903 20:13:31.495711 184992 storage/replica_raft.go:291 [n1,s1,r165/2:/Table/87/1/5/{6-7}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:31.496281 184992 storage/replica_command.go:1430 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r165:/Table/87/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=138] I190903 20:13:31.506900 184992 storage/replica_raft.go:291 [n1,s1,r165/2:/Table/87/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:31.506942 184992 storage/replica_raft.go:304 [n1,s1,r165/2:/Table/87/1/5/{6-7}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:31.507888 184992 storage/replica_command.go:979 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] could not promote [] to voter, rolling back: change replicas of r165 failed: storage/replica_raft.go:305: [n1,s1,r165/2:/Table/87/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:31.507924 184992 storage/queue.go:1027 [n1,replicate,s1,r165/2:/Table/87/1/5/{6-7}] change replicas of r165 failed: storage/replica_raft.go:305: [n1,s1,r165/2:/Table/87/1/5/{6-7}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:31.551571 185093 storage/replica_command.go:1430 [n3,replicate,s3,r103/3:/Table/87/1/3{-/4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r103:/Table/87/1/3{-/4} [(n1,s1):1, (n3,s3):3, next=4, gen=138] I190903 20:13:31.557511 185093 storage/replica_raft.go:291 [n3,s3,r103/3:/Table/87/1/3{-/4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 I190903 20:13:31.565498 185307 storage/store.go:2593 [n1,replicaGC,s1,r103/1:/Table/87/1/3{-/4}] removing replica r103/1 I190903 20:13:31.565976 185307 storage/replica_destroy.go:152 [n1,replicaGC,s1,r103/1:/Table/87/1/3{-/4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:31.602454 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:31.606017 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:31.609739 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) E190903 20:13:31.628033 185410 storage/queue.go:1027 [n1,replicate,s1,r164/2:/Table/87/1/{4-5/6}] no removable replicas from range that needs a removal: [1:0, 2*:27] I190903 20:13:31.764386 185721 storage/replica_command.go:1430 [n1,replicate,s1,r166/3:/Table/87/{1/5/7-2}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r166:/Table/87/{1/5/7-2} [(n3,s3):1, (n1,s1):3, next=4, gen=141] I190903 20:13:31.769895 185721 storage/replica_raft.go:291 [n1,s1,r166/3:/Table/87/{1/5/7-2}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):3] next=4 E190903 20:13:31.773086 184956 storage/queue.go:1027 [n2,replicate,s2,r165/3:/Table/87/1/5/{6-7}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 20:13:31.778993 185879 storage/store.go:2593 [n3,replicaGC,s3,r166/1:/Table/87/{1/5/7-2}] removing replica r166/1 I190903 20:13:31.779402 185879 storage/replica_destroy.go:152 [n3,replicaGC,s3,r166/1:/Table/87/{1/5/7-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:32.128452 186680 storage/replica_command.go:1430 [n2,replicate,s2,r165/3:/Table/87/1/5/{6-7}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r165:/Table/87/1/5/{6-7} [(n1,s1):2, (n2,s2):3, next=4, gen=138] I190903 20:13:32.135496 186680 storage/replica_raft.go:291 [n2,s2,r165/3:/Table/87/1/5/{6-7}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:32.149153 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:32.152504 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 I190903 20:13:32.156857 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:32.163401 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:32.200108 186760 storage/store.go:2593 [n1,replicaGC,s1,r165/2:/Table/87/1/5/{6-7}] removing replica r165/2 I190903 20:13:32.200604 186760 storage/replica_destroy.go:152 [n1,replicaGC,s1,r165/2:/Table/87/1/5/{6-7}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:32.249085 187199 storage/replica_command.go:1430 [n1,replicate,s1,r164/2:/Table/87/1/{4-5/6}] change replicas (add [] remove [(n3,s3):1]): existing descriptor r164:/Table/87/1/{4-5/6} [(n3,s3):1, (n1,s1):2, next=3, gen=134] I190903 20:13:32.253405 187199 storage/replica_raft.go:291 [n1,s1,r164/2:/Table/87/1/{4-5/6}] proposing REMOVE_REPLICA[(n3,s3):1]: after=[(n1,s1):2] next=3 I190903 20:13:32.261570 186955 storage/store.go:2593 [n3,replicaGC,s3,r164/1:/Table/87/1/{4-5/6}] removing replica r164/1 I190903 20:13:32.261896 186955 storage/replica_destroy.go:152 [n3,replicaGC,s3,r164/1:/Table/87/1/{4-5/6}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:32.324832 187425 storage/replica_command.go:609 [n3,merge,s3,r103/3:/Table/87/1/3{-/4}] initiating a merge of r163:/Table/87/1/{3/4-4} [(n3,s3):3, next=4, gen=137] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:32.332745 2255 storage/store.go:2593 [n3,s3,r103/3:/Table/87/1/3{-/4}] removing replica r163/3 I190903 20:13:33.170329 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 4 AND (a, b) < (5, 6) I190903 20:13:33.172568 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 6) AND (a, b) < (5, 7) I190903 20:13:33.177470 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE (a, b) >= (5, 7) I190903 20:13:33.180640 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a < 3 I190903 20:13:33.186732 171865 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "multi col range partitioning - MAXVALUE" WHERE a >= 3 AND a < 4 === RUN TestRepartitioning/single_col_list_partitioning/single_col_range_partitioning I190903 20:13:33.212961 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=87 (multi col range partitioning - MAXVALUE) version=5 mtime=2019-09-03 20:13:33.190312583 +0000 UTC I190903 20:13:33.232674 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 85, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."multi col range partitioning - MAXVALUE"]} I190903 20:13:33.247232 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{multi col range partitioning - MAXVALUE 87 4}] I190903 20:13:33.299967 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=87 (multi col range partitioning - MAXVALUE) version=6 mtime=2019-09-03 20:13:33.299757691 +0000 UTC I190903 20:13:33.361046 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:33.364473 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 88, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:33.376240 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 89, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 20:13:33.377553 190771 storage/replica_command.go:284 [n2,split,s2,r69/2:/{Table/87/2-Max}] initiating a split of this range at key /Table/89 [r70] (zone config) I190903 20:13:33.416152 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:33.420662 191834 storage/replica_command.go:284 [n2,split,s2,r70/1:/{Table/89-Max}] initiating a split of this range at key /Table/89/1 [r141] (zone config) I190903 20:13:33.428497 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:33.441271 192009 storage/replica_command.go:284 [n2,split,s2,r141/1:/{Table/89/1-Max}] initiating a split of this range at key /Table/89/1/3 [r142] (zone config) I190903 20:13:33.441839 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:33.442219 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.533146 192918 storage/replica_command.go:284 [n2,split,s2,r142/1:/{Table/89/1/3-Max}] initiating a split of this range at key /Table/89/1/4 [r143] (zone config) I190903 20:13:33.533232 191984 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r141:/Table/89/1{-/3} [(n2,s2):1, next=2, gen=142] I190903 20:13:33.546863 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.550734 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.556183 191984 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/89/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:33.557343 191984 storage/store_snapshot.go:995 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] sending LEARNER snapshot 04254a5d at applied index 18 I190903 20:13:33.557536 191984 storage/store_snapshot.go:1038 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:33.558842 193064 storage/replica_raftstorage.go:793 [n1,s1,r141/2:{-}] applying LEARNER snapshot [id=04254a5d index=18] I190903 20:13:33.558886 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.560125 193064 storage/replica_raftstorage.go:814 [n1,s1,r141/2:/Table/89/1{-/3}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=04254a5d index=18] I190903 20:13:33.560772 191984 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r141:/Table/89/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=143] I190903 20:13:33.562294 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.569690 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.578953 192880 storage/replica_command.go:284 [n2,split,s2,r143/1:/{Table/89/1/4-Max}] initiating a split of this range at key /Table/89/1/5 [r144] (zone config) I190903 20:13:33.581932 191984 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/89/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:33.581986 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:33.582512 191984 storage/replica_command.go:1430 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r141:/Table/89/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=144] I190903 20:13:33.594449 191984 storage/replica_raft.go:291 [n2,s2,r141/1:/Table/89/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:33.594485 191984 storage/replica_raft.go:304 [n2,s2,r141/1:/Table/89/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:33.595993 191984 storage/replica_command.go:979 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] could not promote [] to voter, rolling back: change replicas of r141 failed: storage/replica_raft.go:305: [n2,s2,r141/1:/Table/89/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:33.596024 191984 storage/queue.go:1027 [n2,replicate,s2,r141/1:/Table/89/1{-/3}] change replicas of r141 failed: storage/replica_raft.go:305: [n2,s2,r141/1:/Table/89/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:33.597752 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:33.602263 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:33.604437 193280 storage/replica_command.go:284 [n2,split,s2,r144/1:/{Table/89/1/5-Max}] initiating a split of this range at key /Table/89/2 [r145] (zone config) I190903 20:13:33.604843 193207 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r143:/Table/89/1/{4-5} [(n2,s2):1, next=2, gen=144] I190903 20:13:33.614592 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:33.617957 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:33.619041 193207 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/89/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:33.621915 193207 storage/store_snapshot.go:995 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] sending LEARNER snapshot e15e863d at applied index 17 I190903 20:13:33.623229 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.624812 193207 storage/store_snapshot.go:1038 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:33.625990 193397 storage/replica_raftstorage.go:793 [n3,s3,r143/2:{-}] applying LEARNER snapshot [id=e15e863d index=17] I190903 20:13:33.627428 193397 storage/replica_raftstorage.go:814 [n3,s3,r143/2:/Table/89/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=e15e863d index=17] I190903 20:13:33.644594 193207 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r143:/Table/89/1/{4-5} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=145] I190903 20:13:33.646014 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.663170 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.666257 193207 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/89/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:33.666812 193207 storage/replica_command.go:1430 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r143:/Table/89/1/{4-5} [(n2,s2):1, (n3,s3):2, next=3, gen=146] I190903 20:13:33.667780 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.685763 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.690577 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.691039 193207 storage/replica_raft.go:291 [n2,s2,r143/1:/Table/89/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:33.691073 193207 storage/replica_raft.go:304 [n2,s2,r143/1:/Table/89/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:33.695807 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.697638 193207 storage/replica_command.go:979 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r143 failed: storage/replica_raft.go:305: [n2,s2,r143/1:/Table/89/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:33.698571 193207 storage/queue.go:1027 [n2,replicate,s2,r143/1:/Table/89/1/{4-5}] change replicas of r143 failed: storage/replica_raft.go:305: [n2,s2,r143/1:/Table/89/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:33.699159 193579 storage/replica_command.go:1430 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r144:/Table/89/{1/5-2} [(n2,s2):1, next=2, gen=145] I190903 20:13:33.706230 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.721211 193579 storage/replica_raft.go:291 [n2,s2,r144/1:/Table/89/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:33.722691 193579 storage/store_snapshot.go:995 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] sending LEARNER snapshot 23ebdd70 at applied index 17 I190903 20:13:33.722861 193579 storage/store_snapshot.go:1038 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:33.723866 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:33.724684 193797 storage/replica_raftstorage.go:793 [n1,s1,r144/2:{-}] applying LEARNER snapshot [id=23ebdd70 index=17] I190903 20:13:33.727361 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:33.733908 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.736111 193797 storage/replica_raftstorage.go:814 [n1,s1,r144/2:/Table/89/{1/5-2}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=23ebdd70 index=17] I190903 20:13:33.736815 193579 storage/replica_command.go:1430 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r144:/Table/89/{1/5-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=146] I190903 20:13:33.742649 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.746435 193579 storage/replica_raft.go:291 [n2,s2,r144/1:/Table/89/{1/5-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:33.747081 193579 storage/replica_command.go:1430 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r144:/Table/89/{1/5-2} [(n2,s2):1, (n1,s1):2, next=3, gen=147] I190903 20:13:33.766974 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.773527 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.786972 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:33.797261 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:33.808370 193579 storage/replica_raft.go:291 [n2,s2,r144/1:/Table/89/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:33.808418 193579 storage/replica_raft.go:304 [n2,s2,r144/1:/Table/89/{1/5-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:33.809225 193579 storage/replica_command.go:979 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] could not promote [] to voter, rolling back: change replicas of r144 failed: storage/replica_raft.go:305: [n2,s2,r144/1:/Table/89/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:33.809258 193579 storage/queue.go:1027 [n2,replicate,s2,r144/1:/Table/89/{1/5-2}] change replicas of r144 failed: storage/replica_raft.go:305: [n2,s2,r144/1:/Table/89/{1/5-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:33.809600 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:33.830540 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.851395 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:33.897788 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:33.970225 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 E190903 20:13:34.107551 194967 storage/queue.go:1027 [n1,replicate,s1,r144/2:/Table/89/{1/5-2}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:34.108215 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:34.128392 194883 storage/replica_command.go:1430 [n1,replicate,s1,r144/2:/Table/89/{1/5-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r144:/Table/89/{1/5-2} [(n2,s2):1, (n1,s1):2, next=3, gen=147] I190903 20:13:34.139216 194883 storage/replica_raft.go:291 [n1,s1,r144/2:/Table/89/{1/5-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:34.147199 194987 storage/store.go:2593 [n2,replicaGC,s2,r144/1:/Table/89/{1/5-2}] removing replica r144/1 I190903 20:13:34.147704 194987 storage/replica_destroy.go:152 [n2,replicaGC,s2,r144/1:/Table/89/{1/5-2}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:34.153825 194944 storage/consistency_queue.go:133 [n2,consistencyChecker,s2,r144/1:/Table/89/{1/5-2}] could not get replica descriptor: r144 was not found on s2 E190903 20:13:34.153862 194944 storage/queue.go:1027 [n2,consistencyChecker,s2,r144/1:/Table/89/{1/5-2}] could not get replica descriptor: r144 was not found on s2 E190903 20:13:34.354455 195655 storage/queue.go:1027 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:34.380560 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:13:34.499091 196078 storage/queue.go:1027 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:34.706734 196611 storage/replica_command.go:1430 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r143:/Table/89/1/{4-5} [(n2,s2):1, (n3,s3):2, next=3, gen=146] I190903 20:13:34.713430 196611 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/89/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 I190903 20:13:34.721229 196619 storage/store.go:2593 [n2,replicaGC,s2,r143/1:/Table/89/1/{4-5}] removing replica r143/1 I190903 20:13:34.721747 196619 storage/replica_destroy.go:152 [n2,replicaGC,s2,r143/1:/Table/89/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:34.758390 196634 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r141:/Table/89/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=144] I190903 20:13:34.763746 196634 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/89/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:34.771915 196854 storage/store.go:2593 [n2,replicaGC,s2,r141/1:/Table/89/1{-/3}] removing replica r141/1 I190903 20:13:34.772358 196854 storage/replica_destroy.go:152 [n2,replicaGC,s2,r141/1:/Table/89/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:34.926585 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:34.928773 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:34.932026 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:34.935716 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:34.939435 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=89 (single col range partitioning) version=2 mtime=2019-09-03 20:13:34.937656018 +0000 UTC I190903 20:13:34.971154 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 89 1}] I190903 20:13:34.972038 197319 sql/lease.go:923 new lease: 89("single col range partitioning") ver=2:1567541927.246722050,0, refcount=0 I190903 20:13:35.031427 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=89 (single col range partitioning) version=3 mtime=2019-09-03 20:13:35.031140741 +0000 UTC I190903 20:13:35.040123 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:35.048395 198153 sql/lease.go:923 new lease: 89("single col range partitioning") ver=3:1567541927.586525431,0, refcount=0 I190903 20:13:35.051476 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 89 2}] I190903 20:13:35.079156 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=89 (single col range partitioning) version=4 mtime=2019-09-03 20:13:35.078298683 +0000 UTC I190903 20:13:35.092215 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 89, info: {TableName:data.public."single col range partitioning" Statement:ALTER TABLE "single col range partitioning" PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:35.096987 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 89 3}] I190903 20:13:35.099110 198531 sql/lease.go:923 new lease: 89("single col range partitioning") ver=4:1567541927.587525431,0, refcount=0 I190903 20:13:35.138199 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:35.139647 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore E190903 20:13:35.143165 199038 storage/queue.go:1027 [n3,merge,s3,r143/2:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:35.153872 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} E190903 20:13:35.160200 198689 storage/queue.go:1027 [n3,merge,s3,r143/2:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:35.171303 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 89, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:35.174230 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 20:13:35.180137 200121 storage/queue.go:1027 [n3,merge,s3,r143/2:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:35.180519 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.183598 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.185629 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.188786 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:35.192458 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.195351 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.197240 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.199187 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.201220 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.207654 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:35.211487 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.215106 200262 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r142:/Table/89/1/{3-4} [(n2,s2):1, next=2, gen=143] I190903 20:13:35.216324 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.219335 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:35.223894 200262 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/89/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:35.225917 200262 storage/store_snapshot.go:995 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] sending LEARNER snapshot a785515d at applied index 23 I190903 20:13:35.226116 200262 storage/store_snapshot.go:1038 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:35.227283 200334 storage/replica_raftstorage.go:793 [n3,s3,r142/2:{-}] applying LEARNER snapshot [id=a785515d index=23] I190903 20:13:35.229097 200334 storage/replica_raftstorage.go:814 [n3,s3,r142/2:/Table/89/1/{3-4}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=a785515d index=23] I190903 20:13:35.229967 200262 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r142:/Table/89/1/{3-4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=144] I190903 20:13:35.230251 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.232144 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.235236 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.238795 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.239914 200262 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/89/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:35.240503 200262 storage/replica_command.go:1430 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r142:/Table/89/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=145] I190903 20:13:35.241529 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.248695 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.254057 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.254672 200262 storage/replica_raft.go:291 [n2,s2,r142/1:/Table/89/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:35.254707 200262 storage/replica_raft.go:304 [n2,s2,r142/1:/Table/89/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:35.255574 200262 storage/replica_command.go:979 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r142 failed: storage/replica_raft.go:305: [n2,s2,r142/1:/Table/89/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:35.255604 200262 storage/queue.go:1027 [n2,replicate,s2,r142/1:/Table/89/1/{3-4}] change replicas of r142 failed: storage/replica_raft.go:305: [n2,s2,r142/1:/Table/89/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:35.257045 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.261356 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.267952 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.285621 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.309889 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.345523 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 E190903 20:13:35.381883 200703 storage/queue.go:1027 [n3,replicate,s3,r142/2:/Table/89/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:30] I190903 20:13:35.414953 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.553112 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.588536 201337 storage/replica_command.go:1430 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r143:/Table/89/1/{4-5} [(n3,s3):2, next=3, gen=147] E190903 20:13:35.588907 201336 storage/queue.go:1027 [n3,merge,s3,r143/2:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:35.595309 201337 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/89/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:35.596897 201337 storage/store_snapshot.go:995 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] sending LEARNER snapshot 4823e571 at applied index 31 I190903 20:13:35.597076 201337 storage/store_snapshot.go:1038 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:35.598297 201057 storage/replica_raftstorage.go:793 [n1,s1,r143/3:{-}] applying LEARNER snapshot [id=4823e571 index=31] I190903 20:13:35.609335 201057 storage/replica_raftstorage.go:814 [n1,s1,r143/3:/Table/89/1/{4-5}] applied LEARNER snapshot [total=11ms ingestion=4@11ms id=4823e571 index=31] I190903 20:13:35.610149 201337 storage/replica_command.go:1430 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r143:/Table/89/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=148] I190903 20:13:35.629625 201337 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/89/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:35.630196 201337 storage/replica_command.go:1430 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r143:/Table/89/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=149] I190903 20:13:35.652152 201337 storage/replica_raft.go:291 [n3,s3,r143/2:/Table/89/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 E190903 20:13:35.652186 201337 storage/replica_raft.go:304 [n3,s3,r143/2:/Table/89/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:35.653790 201337 storage/replica_command.go:979 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r143 failed: storage/replica_raft.go:305: [n3,s3,r143/2:/Table/89/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) E190903 20:13:35.653858 201337 storage/queue.go:1027 [n3,replicate,s3,r143/2:/Table/89/1/{4-5}] change replicas of r143 failed: storage/replica_raft.go:305: [n3,s3,r143/2:/Table/89/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 to remove self (leaseholder) I190903 20:13:35.682747 201546 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r141:/Table/89/1{-/3} [(n1,s1):2, next=3, gen=145] I190903 20:13:35.688331 201546 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/89/1{-/3}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n1,s1):2 (n2,s2):3LEARNER] next=4 I190903 20:13:35.689367 201546 storage/store_snapshot.go:995 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] sending LEARNER snapshot 6a6d7289 at applied index 35 I190903 20:13:35.689576 201546 storage/store_snapshot.go:1038 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:35.690948 201652 storage/replica_raftstorage.go:793 [n2,s2,r141/3:{-}] applying LEARNER snapshot [id=6a6d7289 index=35] I190903 20:13:35.706979 201652 storage/replica_raftstorage.go:814 [n2,s2,r141/3:/Table/89/1{-/3}] applied LEARNER snapshot [total=16ms ingestion=4@16ms id=6a6d7289 index=35] I190903 20:13:35.707516 201546 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] change replicas (add [(n2,s2):3] remove []): existing descriptor r141:/Table/89/1{-/3} [(n1,s1):2, (n2,s2):3LEARNER, next=4, gen=146] I190903 20:13:35.713866 201546 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/89/1{-/3}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n1,s1):2 (n2,s2):3] next=4 I190903 20:13:35.714559 201546 storage/replica_command.go:1430 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r141:/Table/89/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=147] I190903 20:13:35.717137 2222 storage/queue.go:518 [n3,s3,r9/3:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:35.736952 201546 storage/replica_raft.go:291 [n1,s1,r141/2:/Table/89/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 E190903 20:13:35.737016 201546 storage/replica_raft.go:304 [n1,s1,r141/2:/Table/89/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:35.738325 201546 storage/replica_command.go:979 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] could not promote [] to voter, rolling back: change replicas of r141 failed: storage/replica_raft.go:305: [n1,s1,r141/2:/Table/89/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) E190903 20:13:35.738365 201546 storage/queue.go:1027 [n1,replicate,s1,r141/2:/Table/89/1{-/3}] change replicas of r141 failed: storage/replica_raft.go:305: [n1,s1,r141/2:/Table/89/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 to remove self (leaseholder) I190903 20:13:35.828585 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:35.906990 202177 storage/replica_command.go:1430 [n3,replicate,s3,r142/2:/Table/89/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r142:/Table/89/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=145] I190903 20:13:35.921591 202177 storage/replica_raft.go:291 [n3,s3,r142/2:/Table/89/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:35.966601 202439 storage/queue.go:1027 [n2,replicate,s2,r141/3:/Table/89/1{-/3}] no removable replicas from range that needs a removal: [2:0, 3*:44] I190903 20:13:35.988224 201666 storage/store.go:2593 [n2,replicaGC,s2,r142/1:/Table/89/1/{3-4}] removing replica r142/1 I190903 20:13:35.988517 201666 storage/replica_destroy.go:152 [n2,replicaGC,s2,r142/1:/Table/89/1/{3-4}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:36.367559 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:36.371881 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:36.547811 203954 storage/replica_command.go:1430 [n2,replicate,s2,r141/3:/Table/89/1{-/3}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r141:/Table/89/1{-/3} [(n1,s1):2, (n2,s2):3, next=4, gen=147] I190903 20:13:36.555818 203954 storage/replica_raft.go:291 [n2,s2,r141/3:/Table/89/1{-/3}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n2,s2):3] next=4 I190903 20:13:36.564125 204027 storage/store.go:2593 [n1,replicaGC,s1,r141/2:/Table/89/1{-/3}] removing replica r141/2 I190903 20:13:36.564578 204027 storage/replica_destroy.go:152 [n1,replicaGC,s1,r141/2:/Table/89/1{-/3}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] E190903 20:13:36.582854 204117 storage/queue.go:1027 [n3,merge,s3,r143/2:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 E190903 20:13:36.583908 204119 storage/queue.go:1027 [n1,replicate,s1,r143/3:/Table/89/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:40] E190903 20:13:36.807594 204626 storage/queue.go:1027 [n1,merge,s1,r143/3:/Table/89/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):2 I190903 20:13:36.858200 204625 storage/replica_command.go:1430 [n1,replicate,s1,r143/3:/Table/89/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r143:/Table/89/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=149] I190903 20:13:36.866635 204625 storage/replica_raft.go:291 [n1,s1,r143/3:/Table/89/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:36.883005 204665 storage/store.go:2593 [n3,replicaGC,s3,r143/2:/Table/89/1/{4-5}] removing replica r143/2 I190903 20:13:36.883407 204665 storage/replica_destroy.go:152 [n3,replicaGC,s3,r143/2:/Table/89/1/{4-5}] removed 5 (0+5) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:37.178515 205707 storage/replica_command.go:609 [n1,merge,s1,r143/3:/Table/89/1/{4-5}] initiating a merge of r144:/Table/89/{1/5-2} [(n1,s1):2, next=3, gen=148] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:37.185190 212 storage/store.go:2593 [n1,s1,r143/3:/Table/89/1/{4-5}] removing replica r144/2 I190903 20:13:37.375517 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:37.378205 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:37.381379 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:38.440285 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:38.442245 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.445537 189560 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 === RUN TestRepartitioning/single_col_range_partitioning/single_col_list_partitioning I190903 20:13:38.467633 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=89 (single col range partitioning) version=5 mtime=2019-09-03 20:13:38.448922966 +0000 UTC I190903 20:13:38.479940 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 88, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col range partitioning"]} I190903 20:13:38.484705 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col range partitioning 89 4}] I190903 20:13:38.486657 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:38.538793 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=89 (single col range partitioning) version=6 mtime=2019-09-03 20:13:38.538587812 +0000 UTC I190903 20:13:38.558454 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:38.571261 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 90, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:38.585951 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 91, info: {TableName:data.public."single col range partitioning" Statement:CREATE TABLE "single col range partitioning" (a INT8 PRIMARY KEY) PARTITION BY RANGE (a) (PARTITION p3 VALUES FROM (minvalue) TO (3), PARTITION p4 VALUES FROM (3) TO (4)) User:root} I190903 20:13:38.599348 210662 storage/replica_command.go:284 [n2,split,s2,r145/1:/{Table/89/2-Max}] initiating a split of this range at key /Table/91 [r146] (zone config) I190903 20:13:38.683792 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:38.688591 211249 storage/replica_command.go:284 [n2,split,s2,r146/1:/{Table/91-Max}] initiating a split of this range at key /Table/91/1 [r147] (zone config) I190903 20:13:38.711420 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:PARTITION p3 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:38.715871 211751 storage/replica_command.go:284 [n2,split,s2,r147/1:/{Table/91/1-Max}] initiating a split of this range at key /Table/91/1/3 [r148] (zone config) I190903 20:13:38.734652 211900 storage/replica_command.go:284 [n2,split,s2,r148/1:/{Table/91/1/3-Max}] initiating a split of this range at key /Table/91/2 [r149] (zone config) I190903 20:13:38.749116 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:PARTITION p4 OF INDEX data.public."single col range partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:38.749262 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:38.760302 212225 storage/replica_command.go:284 [n2,split,s2,r148/1:/Table/91/{1/3-2}] initiating a split of this range at key /Table/91/1/4 [r150] (zone config) I190903 20:13:38.761610 211858 storage/replica_command.go:1430 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r149:/{Table/91/2-Max} [(n2,s2):1, next=2, gen=149] I190903 20:13:38.771824 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.775720 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.778309 211858 storage/replica_raft.go:291 [n2,s2,r149/1:/{Table/91/2-Max}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:38.798729 211858 storage/store_snapshot.go:995 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] sending LEARNER snapshot b353584d at applied index 14 I190903 20:13:38.800288 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.801345 211858 storage/store_snapshot.go:1038 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.02s I190903 20:13:38.803150 212255 storage/replica_raftstorage.go:793 [n1,s1,r149/2:{-}] applying LEARNER snapshot [id=b353584d index=14] I190903 20:13:38.804606 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.809431 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.814902 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.816361 212255 storage/replica_raftstorage.go:814 [n1,s1,r149/2:/{Table/91/2-Max}] applied LEARNER snapshot [total=12ms ingestion=4@11ms id=b353584d index=14] I190903 20:13:38.816932 211858 storage/replica_command.go:1430 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] change replicas (add [(n1,s1):2] remove []): existing descriptor r149:/{Table/91/2-Max} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=150] I190903 20:13:38.819973 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.831111 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.835374 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:38.843715 211858 storage/replica_raft.go:291 [n2,s2,r149/1:/{Table/91/2-Max}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:38.844748 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:38.847422 211858 storage/replica_command.go:1430 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r149:/{Table/91/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=151] I190903 20:13:38.855229 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.859498 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.862890 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.869735 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.878578 211858 storage/replica_raft.go:291 [n2,s2,r149/1:/{Table/91/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:38.878626 211858 storage/replica_raft.go:304 [n2,s2,r149/1:/{Table/91/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:38.880376 211858 storage/replica_command.go:979 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] could not promote [] to voter, rolling back: change replicas of r149 failed: storage/replica_raft.go:305: [n2,s2,r149/1:/{Table/91/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:38.880444 211858 storage/queue.go:1027 [n2,replicate,s2,r149/1:/{Table/91/2-Max}] change replicas of r149 failed: storage/replica_raft.go:305: [n2,s2,r149/1:/{Table/91/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:38.882477 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.892983 212903 storage/replica_command.go:1430 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r150:/Table/91/{1/4-2} [(n2,s2):1, next=2, gen=150] I190903 20:13:38.896409 212898 storage/replica_command.go:1430 [n1,replicate,s1,r149/2:/{Table/91/2-Max}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r149:/{Table/91/2-Max} [(n2,s2):1, (n1,s1):2, next=3, gen=151] I190903 20:13:38.898546 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 W190903 20:13:38.899852 212860 storage/intentresolver/intent_resolver.go:508 failed to push during intent resolution: failed to push id=6cdb9986 key=/Local/Range/Table/91/1/4/RangeDescriptor rw=true pri=0.06054807 stat=PENDING epo=0 ts=1567541618.892796470,0 orig=0.000000000,0 min=1567541618.892796470,0 max=0.000000000,0 wto=false seq=7 I190903 20:13:38.901264 212903 storage/replica_raft.go:291 [n2,s2,r150/1:/Table/91/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:38.903729 212903 storage/store_snapshot.go:995 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] sending LEARNER snapshot 898e393b at applied index 14 I190903 20:13:38.906681 212903 storage/store_snapshot.go:1038 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:38.909338 212972 storage/replica_raftstorage.go:793 [n1,s1,r150/2:{-}] applying LEARNER snapshot [id=898e393b index=14] I190903 20:13:38.909829 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.915067 212972 storage/replica_raftstorage.go:814 [n1,s1,r150/2:/Table/91/{1/4-2}] applied LEARNER snapshot [total=4ms ingestion=4@1ms id=898e393b index=14] I190903 20:13:38.915604 212903 storage/replica_command.go:1430 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] change replicas (add [(n1,s1):2] remove []): existing descriptor r150:/Table/91/{1/4-2} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=151] I190903 20:13:38.916423 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.916969 212898 storage/replica_raft.go:291 [n1,s1,r149/2:/{Table/91/2-Max}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:38.921278 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.929007 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.941848 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.944915 212903 storage/replica_raft.go:291 [n2,s2,r150/1:/Table/91/{1/4-2}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:38.945658 212903 storage/replica_command.go:1430 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r150:/Table/91/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=152] I190903 20:13:38.950390 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.955674 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.960815 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.964077 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.970131 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.971292 212903 storage/replica_raft.go:291 [n2,s2,r150/1:/Table/91/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:38.971379 212903 storage/replica_raft.go:304 [n2,s2,r150/1:/Table/91/{1/4-2}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:38.972192 212903 storage/replica_command.go:979 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] could not promote [] to voter, rolling back: change replicas of r150 failed: storage/replica_raft.go:305: [n2,s2,r150/1:/Table/91/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:38.972270 212903 storage/queue.go:1027 [n2,replicate,s2,r150/1:/Table/91/{1/4-2}] change replicas of r150 failed: storage/replica_raft.go:305: [n2,s2,r150/1:/Table/91/{1/4-2}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:38.973047 212659 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r148:/Table/91/1/{3-4} [(n2,s2):1, next=2, gen=150] I190903 20:13:38.974259 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.980062 213062 storage/store.go:2593 [n2,replicaGC,s2,r149/1:/{Table/91/2-Max}] removing replica r149/1 I190903 20:13:38.980518 213062 storage/replica_destroy.go:152 [n2,replicaGC,s2,r149/1:/{Table/91/2-Max}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:38.981539 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:38.982429 212659 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/91/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n2,s2):1 (n3,s3):2LEARNER] next=3 I190903 20:13:38.983411 212659 storage/store_snapshot.go:995 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] sending LEARNER snapshot 477d9d5b at applied index 21 I190903 20:13:38.983568 212659 storage/store_snapshot.go:1038 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 11, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:38.984755 213339 storage/replica_raftstorage.go:793 [n3,s3,r148/2:{-}] applying LEARNER snapshot [id=477d9d5b index=21] I190903 20:13:38.988521 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:38.999222 213339 storage/replica_raftstorage.go:814 [n3,s3,r148/2:/Table/91/1/{3-4}] applied LEARNER snapshot [total=14ms ingestion=4@14ms id=477d9d5b index=21] I190903 20:13:39.003116 212659 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] change replicas (add [(n3,s3):2] remove []): existing descriptor r148:/Table/91/1/{3-4} [(n2,s2):1, (n3,s3):2LEARNER, next=3, gen=151] I190903 20:13:39.004055 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.019429 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.020155 212659 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/91/1/{3-4}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n2,s2):1 (n3,s3):2] next=3 I190903 20:13:39.020682 212659 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r148:/Table/91/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=152] I190903 20:13:39.025588 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.032949 212659 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/91/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 E190903 20:13:39.033033 212659 storage/replica_raft.go:304 [n2,s2,r148/1:/Table/91/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:39.033854 212659 storage/replica_command.go:979 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r148 failed: storage/replica_raft.go:305: [n2,s2,r148/1:/Table/91/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:39.033929 212659 storage/queue.go:1027 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] change replicas of r148 failed: storage/replica_raft.go:305: [n2,s2,r148/1:/Table/91/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:39.049849 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.053233 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.059304 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.062944 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.066779 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.073453 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.087871 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.095586 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.100923 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.108856 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.114396 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.126046 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.129136 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.150826 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.155170 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.192644 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:39.268872 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.272111 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 20:13:39.354752 214591 storage/queue.go:1027 [n3,replicate,s3,r148/2:/Table/91/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:28] I190903 20:13:39.409551 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.415639 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 E190903 20:13:39.549027 214633 storage/queue.go:1027 [n1,replicate,s1,r150/2:/Table/91/{1/4-2}] no removable replicas from range that needs a removal: [1:0, 2*:22] I190903 20:13:39.746486 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 3 AND a < 4 I190903 20:13:39.751290 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a >= 4 I190903 20:13:39.755374 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col range partitioning" WHERE a < 3 I190903 20:13:39.760009 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=91 (single col list partitioning) version=2 mtime=2019-09-03 20:13:39.758486079 +0000 UTC I190903 20:13:39.770669 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 91 1}] I190903 20:13:39.771313 215786 sql/lease.go:923 new lease: 91("single col list partitioning") ver=2:1567541905.867441209,0, refcount=0 I190903 20:13:39.829886 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=91 (single col list partitioning) version=3 mtime=2019-09-03 20:13:39.829696345 +0000 UTC I190903 20:13:39.837747 216274 sql/lease.go:923 new lease: 91("single col list partitioning") ver=3:1567541905.868441209,0, refcount=0 I190903 20:13:39.844929 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 91 2}] I190903 20:13:39.870923 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=91 (single col list partitioning) version=4 mtime=2019-09-03 20:13:39.870178047 +0000 UTC I190903 20:13:39.872433 216770 storage/replica_command.go:1430 [n1,replicate,s1,r150/2:/Table/91/{1/4-2}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r150:/Table/91/{1/4-2} [(n2,s2):1, (n1,s1):2, next=3, gen=152] I190903 20:13:39.879116 216770 storage/replica_raft.go:291 [n1,s1,r150/2:/Table/91/{1/4-2}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:39.879328 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 91, info: {TableName:data.public."single col list partitioning" Statement:ALTER TABLE "single col list partitioning" PARTITION BY LIST (a) (PARTITION p3 VALUES IN ((3)), PARTITION p4 VALUES IN ((4))) User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:39.884865 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 91 3}] I190903 20:13:39.887794 216797 sql/lease.go:923 new lease: 91("single col list partitioning") ver=4:1567541920.203623774,0, refcount=0 I190903 20:13:39.891815 217016 storage/store.go:2593 [n2,replicaGC,s2,r150/1:/Table/91/{1/4-2}] removing replica r150/1 I190903 20:13:39.892117 217016 storage/replica_destroy.go:152 [n2,replicaGC,s2,r150/1:/Table/91/{1/4-2}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:39.917771 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:39.919473 217483 storage/replica_command.go:284 [n1,split,s1,r150/2:/Table/91/{1/4-2}] initiating a split of this range at key /Table/91/1/5 [r104] (zone config) I190903 20:13:39.926904 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:39.930962 217100 storage/replica_command.go:1430 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r150:/Table/91/1/{4-5} [(n1,s1):2, next=3, gen=154] I190903 20:13:39.939918 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 91, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:39.940046 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:39.943391 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:39.959320 217100 storage/replica_raft.go:291 [n1,s1,r150/2:/Table/91/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):2 (n3,s3):3LEARNER] next=4 I190903 20:13:39.965900 217100 storage/store_snapshot.go:995 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] sending LEARNER snapshot 58ce9fd3 at applied index 34 I190903 20:13:39.966171 217100 storage/store_snapshot.go:1038 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 14, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:39.966773 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:39.967309 218383 storage/replica_raftstorage.go:793 [n3,s3,r150/3:{-}] applying LEARNER snapshot [id=58ce9fd3 index=34] I190903 20:13:39.968624 218383 storage/replica_raftstorage.go:814 [n3,s3,r150/3:/Table/91/1/{4-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=58ce9fd3 index=34] I190903 20:13:39.969134 217100 storage/replica_command.go:1430 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] change replicas (add [(n3,s3):3] remove []): existing descriptor r150:/Table/91/1/{4-5} [(n1,s1):2, (n3,s3):3LEARNER, next=4, gen=155] I190903 20:13:39.976235 217100 storage/replica_raft.go:291 [n1,s1,r150/2:/Table/91/1/{4-5}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):2 (n3,s3):3] next=4 I190903 20:13:39.976809 217100 storage/replica_command.go:1430 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r150:/Table/91/1/{4-5} [(n1,s1):2, (n3,s3):3, next=4, gen=156] I190903 20:13:39.982274 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:39.986864 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:39.987903 217100 storage/replica_raft.go:291 [n1,s1,r150/2:/Table/91/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 E190903 20:13:39.987991 217100 storage/replica_raft.go:304 [n1,s1,r150/2:/Table/91/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:39.990509 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:39.995666 217100 storage/replica_command.go:979 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r150 failed: storage/replica_raft.go:305: [n1,s1,r150/2:/Table/91/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:39.995704 217100 storage/queue.go:1027 [n1,replicate,s1,r150/2:/Table/91/1/{4-5}] change replicas of r150 failed: storage/replica_raft.go:305: [n1,s1,r150/2:/Table/91/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:39.997463 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:40.001474 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.005655 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:40.011108 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:40.012983 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.024214 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:40.034753 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:40.036752 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:40.040926 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.055293 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.060737 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:40.063073 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.071399 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:40.073505 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.079139 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.087101 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.093656 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.107168 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:40.108944 219213 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] change replicas (add [(n1,s1):2LEARNER] remove []): existing descriptor r147:/Table/91/1{-/3} [(n2,s2):1, next=2, gen=148] I190903 20:13:40.111610 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.119907 219213 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/91/1{-/3}] proposing ADD_REPLICA[(n1,s1):2LEARNER]: after=[(n2,s2):1 (n1,s1):2LEARNER] next=3 I190903 20:13:40.121273 219213 storage/store_snapshot.go:995 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] sending LEARNER snapshot 19f827d2 at applied index 24 I190903 20:13:40.121434 219213 storage/store_snapshot.go:1038 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] streamed snapshot to (n1,s1):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:40.126436 219227 storage/replica_raftstorage.go:793 [n1,s1,r147/2:{-}] applying LEARNER snapshot [id=19f827d2 index=24] I190903 20:13:40.128366 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:40.128955 219227 storage/replica_raftstorage.go:814 [n1,s1,r147/2:/Table/91/1{-/3}] applied LEARNER snapshot [total=2ms ingestion=4@2ms id=19f827d2 index=24] I190903 20:13:40.129597 219213 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] change replicas (add [(n1,s1):2] remove []): existing descriptor r147:/Table/91/1{-/3} [(n2,s2):1, (n1,s1):2LEARNER, next=3, gen=149] I190903 20:13:40.134996 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.137455 219213 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/91/1{-/3}] proposing ADD_REPLICA[(n1,s1):2]: after=[(n2,s2):1 (n1,s1):2] next=3 I190903 20:13:40.138042 219213 storage/replica_command.go:1430 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r147:/Table/91/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=150] I190903 20:13:40.147447 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.149681 219213 storage/replica_raft.go:291 [n2,s2,r147/1:/Table/91/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 E190903 20:13:40.149725 219213 storage/replica_raft.go:304 [n2,s2,r147/1:/Table/91/1{-/3}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:40.150574 219213 storage/replica_command.go:979 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] could not promote [] to voter, rolling back: change replicas of r147 failed: storage/replica_raft.go:305: [n2,s2,r147/1:/Table/91/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) E190903 20:13:40.151361 219213 storage/queue.go:1027 [n2,replicate,s2,r147/1:/Table/91/1{-/3}] change replicas of r147 failed: storage/replica_raft.go:305: [n2,s2,r147/1:/Table/91/1{-/3}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 to remove self (leaseholder) I190903 20:13:40.168042 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 20:13:40.191582 219521 storage/queue.go:1027 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] no removable replicas from range that needs a removal: [1*:33, 2:0] I190903 20:13:40.206861 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.277545 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:40.282347 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.395412 220052 storage/replica_command.go:1430 [n2,replicate,s2,r148/1:/Table/91/1/{3-4}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r148:/Table/91/1/{3-4} [(n2,s2):1, (n3,s3):2, next=3, gen=152] I190903 20:13:40.401404 220052 storage/replica_raft.go:291 [n2,s2,r148/1:/Table/91/1/{3-4}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n2,s2):1] next=3 I190903 20:13:40.410705 219633 storage/store.go:2593 [n3,replicaGC,s3,r148/2:/Table/91/1/{3-4}] removing replica r148/2 I190903 20:13:40.411186 219633 storage/replica_destroy.go:152 [n3,replicaGC,s3,r148/2:/Table/91/1/{3-4}] removed 6 (0+6) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:40.426037 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 E190903 20:13:40.607064 220560 storage/queue.go:1027 [n1,replicate,s1,r147/2:/Table/91/1{-/3}] no removable replicas from range that needs a removal: [1:0, 2*:31] I190903 20:13:40.699360 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:40.719126 298 server/status/runtime.go:498 [n1] runtime stats: 272 MiB RSS, 969 goroutines, 74 MiB/21 MiB/139 MiB GO alloc/idle/total, 57 MiB/115 MiB CGO alloc/total, 6272.0 CGO/sec, 60.4/7.2 %(u/s)time, 0.4 %gc (8x), 100 MiB/100 MiB (r/w)net E190903 20:13:40.839727 221202 storage/queue.go:1027 [n3,replicate,s3,r150/3:/Table/91/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:41] I190903 20:13:40.996410 221716 storage/replica_command.go:1430 [n1,replicate,s1,r147/2:/Table/91/1{-/3}] change replicas (add [] remove [(n2,s2):1]): existing descriptor r147:/Table/91/1{-/3} [(n2,s2):1, (n1,s1):2, next=3, gen=150] I190903 20:13:41.001008 221716 storage/replica_raft.go:291 [n1,s1,r147/2:/Table/91/1{-/3}] proposing REMOVE_REPLICA[(n2,s2):1]: after=[(n1,s1):2] next=3 I190903 20:13:41.008341 221273 storage/store.go:2593 [n2,replicaGC,s2,r147/1:/Table/91/1{-/3}] removing replica r147/1 I190903 20:13:41.008671 221273 storage/replica_destroy.go:152 [n2,replicaGC,s2,r147/1:/Table/91/1{-/3}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:41.285579 222475 storage/replica_command.go:1430 [n3,replicate,s3,r150/3:/Table/91/1/{4-5}] change replicas (add [] remove [(n1,s1):2]): existing descriptor r150:/Table/91/1/{4-5} [(n1,s1):2, (n3,s3):3, next=4, gen=156] I190903 20:13:41.289207 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.290922 222475 storage/replica_raft.go:291 [n3,s3,r150/3:/Table/91/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):2]: after=[(n3,s3):3] next=4 I190903 20:13:41.291169 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.299385 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:41.304072 208954 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:41.305536 221277 storage/store.go:2593 [n1,replicaGC,s1,r150/2:/Table/91/1/{4-5}] removing replica r150/2 === RUN TestRepartitioning/single_col_list_partitioning/unpartitioned I190903 20:13:41.306281 221277 storage/replica_destroy.go:152 [n1,replicaGC,s1,r150/2:/Table/91/1/{4-5}] removed 6 (0+6) keys in 1ms [clear=0ms commit=0ms] I190903 20:13:41.324619 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=91 (single col list partitioning) version=5 mtime=2019-09-03 20:13:41.30596667 +0000 UTC I190903 20:13:41.330390 1776 server/status/runtime.go:498 [n2] runtime stats: 275 MiB RSS, 953 goroutines, 91 MiB/11 MiB/139 MiB GO alloc/idle/total, 58 MiB/117 MiB CGO alloc/total, 5776.0 CGO/sec, 55.7/6.4 %(u/s)time, 0.3 %gc (7x), 100 MiB/100 MiB (r/w)net I190903 20:13:41.343114 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 90, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public."single col list partitioning"]} I190903 20:13:41.362686 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{single col list partitioning 91 4}] I190903 20:13:41.423662 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=91 (single col list partitioning) version=6 mtime=2019-09-03 20:13:41.423445864 +0000 UTC I190903 20:13:41.441513 2001 storage/queue.go:518 [n3,s3] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:41.449553 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 92, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} I190903 20:13:41.456641 2354 server/status/runtime.go:498 [n3] runtime stats: 278 MiB RSS, 958 goroutines, 66 MiB/89 MiB/204 MiB GO alloc/idle/total, 58 MiB/117 MiB CGO alloc/total, 5748.8 CGO/sec, 57.2/6.6 %(u/s)time, 0.3 %gc (8x), 101 MiB/101 MiB (r/w)net I190903 20:13:41.469274 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 93, info: {TableName:data.public."single col list partitioning" Statement:CREATE TABLE "single col list partitioning" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p3 VALUES IN (3), PARTITION p4 VALUES IN (4)) User:root} I190903 20:13:41.471662 224299 storage/replica_command.go:284 [n1,split,s1,r149/2:/{Table/91/2-Max}] initiating a split of this range at key /Table/93 [r105] (zone config) I190903 20:13:41.476691 1812 storage/queue.go:518 [n2,s2] rate limited in MaybeAdd (split): throttled on async limiting semaphore I190903 20:13:41.499144 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 93, info: {Target:INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:41.504962 224838 storage/replica_command.go:284 [n1,split,s1,r105/1:/{Table/93-Max}] initiating a split of this range at key /Table/93/1 [r106] (zone config) I190903 20:13:41.511783 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 93, info: {Target:PARTITION p3 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:41.529669 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 93, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:41.530263 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.536216 1684 storage/queue.go:518 [n2,s2,r3/3:/System/{NodeLive…-tsd}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:41.543627 225248 storage/replica_command.go:284 [n1,split,s1,r106/1:/{Table/93/1-Max}] initiating a split of this range at key /Table/93/1/3 [r107] (zone config) I190903 20:13:41.560225 225891 storage/replica_command.go:284 [n1,split,s1,r107/1:/{Table/93/1/3-Max}] initiating a split of this range at key /Table/93/1/4 [r108] (zone config) I190903 20:13:41.577005 225912 storage/replica_command.go:284 [n1,split,s1,r108/1:/{Table/93/1/4-Max}] initiating a split of this range at key /Table/93/1/5 [r109] (zone config) I190903 20:13:41.577519 226026 storage/replica_command.go:1430 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r107:/Table/93/1/{3-4} [(n1,s1):1, next=2, gen=156] I190903 20:13:41.591614 226026 storage/replica_raft.go:291 [n1,s1,r107/1:/Table/93/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:13:41.593331 226026 storage/store_snapshot.go:995 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] sending LEARNER snapshot ca5f77ce at applied index 17 I190903 20:13:41.593516 226026 storage/store_snapshot.go:1038 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:41.593736 225953 storage/replica_command.go:284 [n1,split,s1,r109/1:/{Table/93/1/5-Max}] initiating a split of this range at key /Table/93/2 [r110] (zone config) I190903 20:13:41.594503 226055 storage/replica_raftstorage.go:793 [n2,s2,r107/2:{-}] applying LEARNER snapshot [id=ca5f77ce index=17] I190903 20:13:41.595677 226055 storage/replica_raftstorage.go:814 [n2,s2,r107/2:/Table/93/1/{3-4}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=ca5f77ce index=17] I190903 20:13:41.596156 226026 storage/replica_command.go:1430 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] change replicas (add [(n2,s2):2] remove []): existing descriptor r107:/Table/93/1/{3-4} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=157] I190903 20:13:41.620468 226026 storage/replica_raft.go:291 [n1,s1,r107/1:/Table/93/1/{3-4}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:13:41.622590 226026 storage/replica_command.go:1430 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r107:/Table/93/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=158] I190903 20:13:41.624184 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.627445 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.631072 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.635372 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.638643 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.642010 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.645940 226026 storage/replica_raft.go:291 [n1,s1,r107/1:/Table/93/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:41.646007 226026 storage/replica_raft.go:304 [n1,s1,r107/1:/Table/93/1/{3-4}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:41.650242 226026 storage/replica_command.go:979 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] could not promote [] to voter, rolling back: change replicas of r107 failed: storage/replica_raft.go:305: [n1,s1,r107/1:/Table/93/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:41.650273 226026 storage/queue.go:1027 [n1,replicate,s1,r107/1:/Table/93/1/{3-4}] change replicas of r107 failed: storage/replica_raft.go:305: [n1,s1,r107/1:/Table/93/1/{3-4}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:41.651209 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.652131 225923 storage/replica_command.go:1430 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r108:/Table/93/1/{4-5} [(n1,s1):1, next=2, gen=157] I190903 20:13:41.658103 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.661363 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.664025 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.665713 225923 storage/replica_raft.go:291 [n1,s1,r108/1:/Table/93/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:41.667622 225923 storage/store_snapshot.go:995 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] sending LEARNER snapshot 01a915ff at applied index 17 I190903 20:13:41.667885 225923 storage/store_snapshot.go:1038 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:41.670805 226012 storage/replica_raftstorage.go:793 [n3,s3,r108/2:{-}] applying LEARNER snapshot [id=01a915ff index=17] I190903 20:13:41.673617 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.679442 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.680985 226012 storage/replica_raftstorage.go:814 [n3,s3,r108/2:/Table/93/1/{4-5}] applied LEARNER snapshot [total=10ms ingestion=4@10ms id=01a915ff index=17] I190903 20:13:41.682165 225923 storage/replica_command.go:1430 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r108:/Table/93/1/{4-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=158] I190903 20:13:41.682971 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.687233 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:41.687576 225923 storage/replica_raft.go:291 [n1,s1,r108/1:/Table/93/1/{4-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:41.688049 225923 storage/replica_command.go:1430 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r108:/Table/93/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=159] I190903 20:13:41.702770 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.704660 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.706892 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:41.710902 225923 storage/replica_raft.go:291 [n1,s1,r108/1:/Table/93/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:41.710938 225923 storage/replica_raft.go:304 [n1,s1,r108/1:/Table/93/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:41.710964 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.713861 225923 storage/replica_command.go:979 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r108 failed: storage/replica_raft.go:305: [n1,s1,r108/1:/Table/93/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:41.713923 225923 storage/queue.go:1027 [n1,replicate,s1,r108/1:/Table/93/1/{4-5}] change replicas of r108 failed: storage/replica_raft.go:305: [n1,s1,r108/1:/Table/93/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:41.714697 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.716704 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.720025 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.726262 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.732017 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.735149 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:41.739196 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.742557 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.745873 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.785978 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:41.787992 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.795713 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.801270 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.814032 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.819480 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.821335 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.824013 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.828269 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.834580 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.838727 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.843691 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.849769 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.852442 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.859270 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.877382 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:41.882622 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.884315 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.903220 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.905453 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:41.942910 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:41.944916 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:41.947809 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:42.018847 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:42.157557 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:42.159610 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 E190903 20:13:42.226655 227703 storage/queue.go:1027 [n2,replicate,s2,r107/2:/Table/93/1/{3-4}] no removable replicas from range that needs a removal: [1:0, 2*:24] I190903 20:13:42.410174 228155 storage/replica_command.go:1430 [n2,replicate,s2,r107/2:/Table/93/1/{3-4}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r107:/Table/93/1/{3-4} [(n1,s1):1, (n2,s2):2, next=3, gen=158] I190903 20:13:42.416548 228155 storage/replica_raft.go:291 [n2,s2,r107/2:/Table/93/1/{3-4}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:13:42.425650 228330 storage/store.go:2593 [n1,replicaGC,s1,r107/1:/Table/93/1/{3-4}] removing replica r107/1 I190903 20:13:42.426062 228330 storage/replica_destroy.go:152 [n1,replicaGC,s1,r107/1:/Table/93/1/{3-4}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:42.429892 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:42.431827 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:42.487524 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 E190903 20:13:42.692581 229026 storage/queue.go:1027 [n3,replicate,s3,r108/2:/Table/93/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:43.026720 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 3 I190903 20:13:43.030239 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a = 4 I190903 20:13:43.034085 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a > 4 I190903 20:13:43.035819 220929 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning" WHERE a < 3 I190903 20:13:43.039087 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=93 (unpartitioned) version=2 mtime=2019-09-03 20:13:43.037742195 +0000 UTC I190903 20:13:43.051318 230011 sql/lease.go:923 new lease: 93("unpartitioned") ver=2:1567541935.567424491,0, refcount=0 I190903 20:13:43.054056 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 93 1}] I190903 20:13:43.061925 230490 storage/replica_command.go:1430 [n3,replicate,s3,r108/2:/Table/93/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r108:/Table/93/1/{4-5} [(n1,s1):1, (n3,s3):2, next=3, gen=159] I190903 20:13:43.068024 230490 storage/replica_raft.go:291 [n3,s3,r108/2:/Table/93/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:43.079764 229983 storage/store.go:2593 [n1,replicaGC,s1,r108/1:/Table/93/1/{4-5}] removing replica r108/1 I190903 20:13:43.080682 229983 storage/replica_destroy.go:152 [n1,replicaGC,s1,r108/1:/Table/93/1/{4-5}] removed 5 (0+5) keys in 1ms [clear=1ms commit=0ms] I190903 20:13:43.104377 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=93 (unpartitioned) version=3 mtime=2019-09-03 20:13:43.104181835 +0000 UTC I190903 20:13:43.130363 230659 sql/lease.go:923 new lease: 93("unpartitioned") ver=3:1567541935.568424491,0, refcount=0 I190903 20:13:43.130688 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 93 2}] I190903 20:13:43.159425 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=93 (unpartitioned) version=4 mtime=2019-09-03 20:13:43.154895888 +0000 UTC I190903 20:13:43.172137 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 93, info: {TableName:data.public.unpartitioned Statement:ALTER TABLE unpartitioned PARTITION BY NOTHING User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:43.180368 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 93 3}] E190903 20:13:43.180541 231410 storage/queue.go:1027 [n2,merge,s2,r107/2:/Table/93/1/{3-4}] none of the remaining targets [(n2,s2):?] are legal additions to (n3,s3):2 I190903 20:13:43.181418 231083 storage/replica_command.go:1430 [n2,s2,r107/2:/Table/93/1/{3-4}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r107:/Table/93/1/{3-4} [(n2,s2):2, next=3, gen=159] I190903 20:13:43.188137 230163 sql/lease.go:923 new lease: 93("unpartitioned") ver=4:1567541935.569424491,0, refcount=0 E190903 20:13:43.188234 231445 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.194130 231083 storage/replica_raft.go:291 [n2,s2,r107/2:/Table/93/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:43.199553 231083 storage/store_snapshot.go:995 [n2,s2,r107/2:/Table/93/1/{3-4}] sending LEARNER snapshot 0f53f97c at applied index 34 I190903 20:13:43.199747 231083 storage/store_snapshot.go:1038 [n2,s2,r107/2:/Table/93/1/{3-4}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 13, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:43.203367 231685 storage/replica_raftstorage.go:793 [n1,s1,r107/3:{-}] applying LEARNER snapshot [id=0f53f97c index=34] I190903 20:13:43.217071 231685 storage/replica_raftstorage.go:814 [n1,s1,r107/3:/Table/93/1/{3-4}] applied LEARNER snapshot [total=13ms ingestion=4@12ms id=0f53f97c index=34] I190903 20:13:43.217644 231083 storage/replica_command.go:1430 [n2,s2,r107/2:/Table/93/1/{3-4}] change replicas (add [(n1,s1):3] remove []): existing descriptor r107:/Table/93/1/{3-4} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=160] === RUN TestRepartitioning/single_col_list_partitioning_-_DEFAULT/unpartitioned I190903 20:13:43.261677 231083 storage/replica_raft.go:291 [n2,s2,r107/2:/Table/93/1/{3-4}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 E190903 20:13:43.267415 231819 storage/queue.go:1027 [n1,replicate,s1,r107/3:/Table/93/1/{3-4}] no removable replicas from range that needs a removal: [2:0, 3*:38] I190903 20:13:43.282631 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=93 (unpartitioned) version=5 mtime=2019-09-03 20:13:43.244257578 +0000 UTC I190903 20:13:43.299990 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 92, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 20:13:43.309613 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 93 4}] E190903 20:13:43.318080 232171 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.362587 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=93 (unpartitioned) version=6 mtime=2019-09-03 20:13:43.362407711 +0000 UTC E190903 20:13:43.392633 232862 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.392976 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_database", target: 94, info: {DatabaseName:data Statement:CREATE DATABASE data User:root} E190903 20:13:43.396938 233391 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.402501 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "create_table", target: 95, info: {TableName:data.public."single col list partitioning - DEFAULT" Statement:CREATE TABLE "single col list partitioning - DEFAULT" (a INT8 PRIMARY KEY) PARTITION BY LIST (a) (PARTITION p4 VALUES IN (4), PARTITION p5 VALUES IN (5), PARTITION pd VALUES IN (DEFAULT)) User:root} I190903 20:13:43.411005 225646 storage/replica_command.go:284 [n1,split,s1,r110/1:/{Table/93/2-Max}] initiating a split of this range at key /Table/95 [r191] (zone config) I190903 20:13:43.415450 232572 storage/replica_command.go:1430 [n1,s1,r107/3:/Table/93/1/{3-4}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r107:/Table/93/1/{3-4} [(n2,s2):2, (n1,s1):3, next=4, gen=161] E190903 20:13:43.420178 233643 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.425898 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 95, info: {Target:PARTITION p4 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n2]' User:root} I190903 20:13:43.427854 233886 storage/replica_command.go:284 [n1,split,s1,r191/1:/{Table/95-Max}] initiating a split of this range at key /Table/95/1/4 [r192] (zone config) I190903 20:13:43.435798 232572 storage/replica_raft.go:291 [n1,s1,r107/3:/Table/93/1/{3-4}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:43.436782 234257 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.437405 234102 storage/replica_command.go:284 [n1,split,s1,r192/1:/{Table/95/1/4-Max}] initiating a split of this range at key /Table/95/1/5 [r193] (zone config) I190903 20:13:43.437940 234267 storage/replica_command.go:1430 [n1,replicate,s1,r191/1:/Table/95{-/1/4}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r191:/Table/95{-/1/4} [(n1,s1):1, next=2, gen=160] I190903 20:13:43.439075 231366 storage/replica_command.go:609 [n1,merge,s1,r106/1:/Table/93/1{-/3}] initiating a merge of r107:/Table/93/1/{3-4} [(n1,s1):3, next=4, gen=162] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:43.447466 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 95, info: {Target:PARTITION p5 OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n3]' User:root} I190903 20:13:43.448707 219 storage/store.go:2593 [n1,s1,r106/1:/Table/93/1{-/3}] removing replica r107/3 I190903 20:13:43.449061 234411 storage/store.go:2593 [n2,replicaGC,s2,r107/2:/Table/93/1/{3-4}] removing replica r107/2 I190903 20:13:43.450003 234411 storage/replica_destroy.go:152 [n2,replicaGC,s2,r107/2:/Table/93/1/{3-4}] removed 6 (0+6) keys in 1ms [clear=1ms commit=0ms] I190903 20:13:43.454375 234267 storage/replica_raft.go:291 [n1,s1,r191/1:/Table/95{-/1/4}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 E190903 20:13:43.454916 234684 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] none of the remaining targets [(n3,s3):?] are legal additions to (n1,s1):1 I190903 20:13:43.458162 234112 storage/replica_command.go:1430 [n3,s3,r108/2:/Table/93/1/{4-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r108:/Table/93/1/{4-5} [(n3,s3):2, next=3, gen=160] I190903 20:13:43.465820 234783 storage/replica_command.go:284 [n1,split,s1,r193/1:/{Table/95/1/5-Max}] initiating a split of this range at key /Table/95/1/6 [r194] (zone config) I190903 20:13:43.468214 234112 storage/replica_raft.go:291 [n3,s3,r108/2:/Table/93/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n3,s3):2 (n1,s1):3LEARNER] next=4 I190903 20:13:43.470925 234112 storage/store_snapshot.go:995 [n3,s3,r108/2:/Table/93/1/{4-5}] sending LEARNER snapshot 4580ef77 at applied index 31 I190903 20:13:43.471093 234112 storage/store_snapshot.go:1038 [n3,s3,r108/2:/Table/93/1/{4-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 12, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.473190 234267 storage/store_snapshot.go:995 [n1,replicate,s1,r191/1:/Table/95{-/1/4}] sending LEARNER snapshot 8bfeb9cd at applied index 17 I190903 20:13:43.473340 234267 storage/store_snapshot.go:1038 [n1,replicate,s1,r191/1:/Table/95{-/1/4}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.475148 234447 storage/replica_raftstorage.go:793 [n1,s1,r108/3:{-}] applying LEARNER snapshot [id=4580ef77 index=31] I190903 20:13:43.478349 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "set_zone_config", target: 95, info: {Target:PARTITION pd OF INDEX data.public."single col list partitioning - DEFAULT"@primary Config: Options:constraints = '[+n1]' User:root} I190903 20:13:43.478901 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.492611 234426 storage/replica_raftstorage.go:793 [n3,s3,r191/2:{-}] applying LEARNER snapshot [id=8bfeb9cd index=17] I190903 20:13:43.493646 234447 storage/replica_raftstorage.go:814 [n1,s1,r108/3:/Table/93/1/{4-5}] applied LEARNER snapshot [total=18ms ingestion=4@18ms id=4580ef77 index=31] I190903 20:13:43.494207 234895 storage/replica_command.go:284 [n1,split,s1,r191/1:/Table/95{-/1/4}] initiating a split of this range at key /Table/95/1 [r195] (zone config); r191/2 is being probed (may or may not need a Raft snapshot) I190903 20:13:43.496698 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.500763 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.501074 234112 storage/replica_command.go:1430 [n3,s3,r108/2:/Table/93/1/{4-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=161] I190903 20:13:43.502158 235092 storage/replica_command.go:1430 [n3,s3,r108/2:/Table/93/1/{4-5}] change replicas (add [] remove [(n1,s1):3LEARNER]): existing descriptor r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=161] I190903 20:13:43.508949 235098 storage/replica_command.go:284 [n1,split,s1,r194/1:/{Table/95/1/6-Max}] initiating a split of this range at key /Table/95/2 [r196] (zone config) I190903 20:13:43.509511 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.513992 234426 storage/replica_raftstorage.go:814 [n3,s3,r191/2:/Table/95{-/1/4}] applied LEARNER snapshot [total=21ms ingestion=4@15ms id=8bfeb9cd index=17] I190903 20:13:43.515860 234267 storage/replica_command.go:979 [n1,replicate,s1,r191/1:/Table/95{-/1}] could not promote [n3,s3] to voter, rolling back: change replicas of r191 failed: descriptor changed: [expected] r191:/Table/95{-/1/4} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=161] != [actual] r191:/Table/95{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=162] I190903 20:13:43.516046 234267 storage/replica_command.go:1430 [n1,replicate,s1,r191/1:/Table/95{-/1}] change replicas (add [] remove [(n3,s3):2LEARNER]): existing descriptor r191:/Table/95{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=162] I190903 20:13:43.516552 234112 storage/replica_raft.go:291 [n3,s3,r108/2:/Table/93/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n3,s3):2 (n1,s1):3] next=4 I190903 20:13:43.526879 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.527928 234267 storage/replica_raft.go:291 [n1,s1,r191/1:/Table/95{-/1}] proposing REMOVE_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1] next=3 I190903 20:13:43.528311 234267 storage/replica_command.go:1217 [n1,replicate,s1,r191/1:/Table/95{-/1}] rolled back learner n3,s3 in r191:/Table/95{-/1} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=162] I190903 20:13:43.528819 235592 storage/replica_command.go:1430 [n1,replicate,s1,r195/1:/Table/95/1{-/4}] change replicas (add [] remove [(n3,s3):2LEARNER]): existing descriptor r195:/Table/95/1{-/4} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=162] I190903 20:13:43.529175 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.530941 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.557316 235392 storage/store.go:2593 [n3,replicaGC,s3,r191/2:/Table/95{-/1}] removing replica r191/2 I190903 20:13:43.557710 235392 storage/replica_destroy.go:152 [n3,replicaGC,s3,r191/2:/Table/95{-/1}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:43.560684 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.562947 235092 storage/replica_command.go:979 [n3,s3,r108/2:/Table/93/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r108 failed: descriptor changed: [expected] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=161] != [actual] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] E190903 20:13:43.563106 235092 storage/queue.go:1027 [n3,merge,s3,r108/2:/Table/93/1/{4-5}] removing learners from r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=161]: change replicas of r108 failed: descriptor changed: [expected] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3LEARNER, next=4, gen=161] != [actual] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] I190903 20:13:43.564359 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.567559 235592 storage/replica_raft.go:291 [n1,s1,r195/1:/Table/95/1{-/4}] proposing REMOVE_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1] next=3 I190903 20:13:43.571264 235315 storage/replica_command.go:1430 [n1,replicate,s1,r108/3:/Table/93/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] I190903 20:13:43.571283 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.572696 235521 storage/replica_command.go:1430 [n1,s1,r108/3:/Table/93/1/{4-5}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] I190903 20:13:43.576412 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.583905 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.583990 235315 storage/replica_raft.go:291 [n1,s1,r108/3:/Table/93/1/{4-5}] proposing REMOVE_REPLICA[(n3,s3):2]: after=[(n1,s1):3] next=4 I190903 20:13:43.585157 235537 storage/replica_command.go:1430 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] change replicas (add [(n2,s2):2LEARNER] remove []): existing descriptor r192:/Table/95/1/{4-5} [(n1,s1):1, next=2, gen=161] I190903 20:13:43.594805 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.595176 235521 storage/replica_command.go:979 [n1,s1,r108/3:/Table/93/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r108 failed: descriptor changed: [expected] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] != [actual] r108:/Table/93/1/{4-5} [(n1,s1):3, next=4, gen=163] W190903 20:13:43.595385 234112 storage/replica_command.go:2033 [n3,s3,r108/2:/Table/93/1/{4-5}] while removing target n3,s3: change replicas of r108 failed: descriptor changed: [expected] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] != [actual] r108:/Table/93/1/{4-5} [(n1,s1):3, next=4, gen=163] E190903 20:13:43.595561 234111 storage/queue.go:1027 [n1,merge,s1,r106/1:/Table/93/1{-/4}] change replicas of r108 failed: descriptor changed: [expected] r108:/Table/93/1/{4-5} [(n3,s3):2, (n1,s1):3, next=4, gen=162] != [actual] r108:/Table/93/1/{4-5} [(n1,s1):3, next=4, gen=163] I190903 20:13:43.596615 235782 storage/replica_command.go:609 [n1,merge,s1,r106/1:/Table/93/1{-/4}] initiating a merge of r108:/Table/93/1/{4-5} [(n1,s1):3, next=4, gen=163] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:43.602916 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.606084 235537 storage/replica_raft.go:291 [n1,s1,r192/1:/Table/95/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2LEARNER]: after=[(n1,s1):1 (n2,s2):2LEARNER] next=3 I190903 20:13:43.607308 235537 storage/store_snapshot.go:995 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] sending LEARNER snapshot fd1035fa at applied index 18 I190903 20:13:43.607453 235537 storage/store_snapshot.go:1038 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] streamed snapshot to (n2,s2):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.607700 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.608201 220 storage/store.go:2593 [n1,s1,r106/1:/Table/93/1{-/4}] removing replica r108/3 I190903 20:13:43.609106 235707 storage/replica_raftstorage.go:793 [n2,s2,r192/2:{-}] applying LEARNER snapshot [id=fd1035fa index=18] I190903 20:13:43.610768 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.610941 235707 storage/replica_raftstorage.go:814 [n2,s2,r192/2:/Table/95/1/{4-5}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=fd1035fa index=18] I190903 20:13:43.611366 235537 storage/replica_command.go:1430 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] change replicas (add [(n2,s2):2] remove []): existing descriptor r192:/Table/95/1/{4-5} [(n1,s1):1, (n2,s2):2LEARNER, next=3, gen=162] I190903 20:13:43.615099 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.617235 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.622859 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.624822 235537 storage/replica_raft.go:291 [n1,s1,r192/1:/Table/95/1/{4-5}] proposing ADD_REPLICA[(n2,s2):2]: after=[(n1,s1):1 (n2,s2):2] next=3 I190903 20:13:43.625027 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.625636 235537 storage/replica_command.go:1430 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r192:/Table/95/1/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=163] I190903 20:13:43.626873 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.629562 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.632845 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.636859 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.640932 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.641086 235657 storage/store.go:2593 [n3,replicaGC,s3,r108/2:/Table/93/1/{4-5}] removing replica r108/2 I190903 20:13:43.641596 235657 storage/replica_destroy.go:152 [n3,replicaGC,s3,r108/2:/Table/93/1/{4-5}] removed 8 (0+8) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:43.642476 235537 storage/replica_raft.go:291 [n1,s1,r192/1:/Table/95/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 E190903 20:13:43.642523 235537 storage/replica_raft.go:304 [n1,s1,r192/1:/Table/95/1/{4-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:43.643469 235537 storage/replica_command.go:979 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] could not promote [] to voter, rolling back: change replicas of r192 failed: storage/replica_raft.go:305: [n1,s1,r192/1:/Table/95/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) E190903 20:13:43.643504 235537 storage/queue.go:1027 [n1,replicate,s1,r192/1:/Table/95/1/{4-5}] change replicas of r192 failed: storage/replica_raft.go:305: [n1,s1,r192/1:/Table/95/1/{4-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 to remove self (leaseholder) I190903 20:13:43.643828 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.644030 234130 storage/replica_command.go:1430 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r193:/Table/95/1/{5-6} [(n1,s1):1, next=2, gen=162] I190903 20:13:43.646505 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:43.649119 234130 storage/replica_raft.go:291 [n1,s1,r193/1:/Table/95/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:43.650071 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.650589 234130 storage/store_snapshot.go:995 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] sending LEARNER snapshot 3a38fc57 at applied index 18 I190903 20:13:43.650759 234130 storage/store_snapshot.go:1038 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.652056 235738 storage/replica_raftstorage.go:793 [n3,s3,r193/2:{-}] applying LEARNER snapshot [id=3a38fc57 index=18] I190903 20:13:43.653577 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.654606 235738 storage/replica_raftstorage.go:814 [n3,s3,r193/2:/Table/95/1/{5-6}] applied LEARNER snapshot [total=3ms ingestion=4@2ms id=3a38fc57 index=18] I190903 20:13:43.655790 234130 storage/replica_command.go:1430 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] change replicas (add [(n3,s3):2] remove []): existing descriptor r193:/Table/95/1/{5-6} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=163] I190903 20:13:43.656394 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.660569 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.663305 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.665312 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.668059 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.669908 234130 storage/replica_raft.go:291 [n1,s1,r193/1:/Table/95/1/{5-6}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:43.670453 234130 storage/replica_command.go:1430 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r193:/Table/95/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=164] I190903 20:13:43.678719 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.682863 236136 storage/replica_command.go:609 [n1,merge,s1,r106/1:/Table/93/1{-/5}] initiating a merge of r109:/Table/93/{1/5-2} [(n1,s1):1, next=2, gen=158] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:43.682890 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:43.686058 234130 storage/replica_raft.go:291 [n1,s1,r193/1:/Table/95/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:43.686102 234130 storage/replica_raft.go:304 [n1,s1,r193/1:/Table/95/1/{5-6}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:43.687495 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.689249 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.692461 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.692755 234130 storage/replica_command.go:979 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] could not promote [] to voter, rolling back: change replicas of r193 failed: storage/replica_raft.go:305: [n1,s1,r193/1:/Table/95/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:43.692812 234130 storage/queue.go:1027 [n1,replicate,s1,r193/1:/Table/95/1/{5-6}] change replicas of r193 failed: storage/replica_raft.go:305: [n1,s1,r193/1:/Table/95/1/{5-6}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:43.693874 236109 storage/replica_command.go:1430 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r196:/{Table/95/2-Max} [(n1,s1):1, next=2, gen=163] I190903 20:13:43.699106 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.699159 236109 storage/replica_raft.go:291 [n1,s1,r196/1:/{Table/95/2-Max}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:43.700578 236109 storage/store_snapshot.go:995 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] sending LEARNER snapshot 17028069 at applied index 14 I190903 20:13:43.700724 236109 storage/store_snapshot.go:1038 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 7, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.701824 236093 storage/replica_raftstorage.go:793 [n3,s3,r196/2:{-}] applying LEARNER snapshot [id=17028069 index=14] I190903 20:13:43.703010 193 storage/store.go:2593 [n1,s1,r106/1:/Table/93/1{-/5}] removing replica r109/1 I190903 20:13:43.703381 236093 storage/replica_raftstorage.go:814 [n3,s3,r196/2:/{Table/95/2-Max}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=17028069 index=14] I190903 20:13:43.703810 236109 storage/replica_command.go:1430 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] change replicas (add [(n3,s3):2] remove []): existing descriptor r196:/{Table/95/2-Max} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=164] I190903 20:13:43.705969 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.708402 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.736973 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.737185 236109 storage/replica_raft.go:291 [n1,s1,r196/1:/{Table/95/2-Max}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:43.737960 236109 storage/replica_command.go:1430 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r196:/{Table/95/2-Max} [(n1,s1):1, (n3,s3):2, next=3, gen=165] I190903 20:13:43.739098 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.746950 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.750058 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.750306 236109 storage/replica_raft.go:291 [n1,s1,r196/1:/{Table/95/2-Max}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:43.750351 236109 storage/replica_raft.go:304 [n1,s1,r196/1:/{Table/95/2-Max}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:43.751144 236109 storage/replica_command.go:979 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] could not promote [] to voter, rolling back: change replicas of r196 failed: storage/replica_raft.go:305: [n1,s1,r196/1:/{Table/95/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:43.751190 236109 storage/queue.go:1027 [n1,replicate,s1,r196/1:/{Table/95/2-Max}] change replicas of r196 failed: storage/replica_raft.go:305: [n1,s1,r196/1:/{Table/95/2-Max}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:43.753099 236239 storage/replica_command.go:1430 [n1,replicate,s1,r191/1:/Table/95{-/1}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r191:/Table/95{-/1} [(n1,s1):1, next=3, gen=163] E190903 20:13:43.755583 236364 storage/queue.go:1027 [n2,replicate,s2,r192/2:/Table/95/1/{4-5}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:43.761282 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.771861 236239 storage/replica_raft.go:291 [n1,s1,r191/1:/Table/95{-/1}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 20:13:43.774091 236239 storage/store_snapshot.go:995 [n1,replicate,s1,r191/1:/Table/95{-/1}] sending LEARNER snapshot 07342fb1 at applied index 25 I190903 20:13:43.774306 236239 storage/store_snapshot.go:1038 [n1,replicate,s1,r191/1:/Table/95{-/1}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 15, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.775537 236436 storage/replica_raftstorage.go:793 [n3,s3,r191/3:{-}] applying LEARNER snapshot [id=07342fb1 index=25] I190903 20:13:43.793329 236436 storage/replica_raftstorage.go:814 [n3,s3,r191/3:/Table/95{-/1}] applied LEARNER snapshot [total=18ms ingestion=4@17ms id=07342fb1 index=25] I190903 20:13:43.793952 236239 storage/replica_command.go:1430 [n1,replicate,s1,r191/1:/Table/95{-/1}] change replicas (add [(n3,s3):3] remove []): existing descriptor r191:/Table/95{-/1} [(n1,s1):1, (n3,s3):3LEARNER, next=4, gen=164] I190903 20:13:43.800038 236239 storage/replica_raft.go:291 [n1,s1,r191/1:/Table/95{-/1}] proposing ADD_REPLICA[(n3,s3):3]: after=[(n1,s1):1 (n3,s3):3] next=4 I190903 20:13:43.800586 236239 storage/replica_command.go:1430 [n1,replicate,s1,r191/1:/Table/95{-/1}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r191:/Table/95{-/1} [(n1,s1):1, (n3,s3):3, next=4, gen=165] I190903 20:13:43.814034 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:13:43.814203 236239 storage/replica_raft.go:291 [n1,s1,r191/1:/Table/95{-/1}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 E190903 20:13:43.814250 236239 storage/replica_raft.go:304 [n1,s1,r191/1:/Table/95{-/1}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:43.816613 236239 storage/replica_command.go:979 [n1,replicate,s1,r191/1:/Table/95{-/1}] could not promote [] to voter, rolling back: change replicas of r191 failed: storage/replica_raft.go:305: [n1,s1,r191/1:/Table/95{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) E190903 20:13:43.816684 236239 storage/queue.go:1027 [n1,replicate,s1,r191/1:/Table/95{-/1}] change replicas of r191 failed: storage/replica_raft.go:305: [n1,s1,r191/1:/Table/95{-/1}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):3] next=4 to remove self (leaseholder) I190903 20:13:43.817190 236126 storage/replica_command.go:1430 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] change replicas (add [(n2,s2):4LEARNER] remove []): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):3, next=4, gen=55] I190903 20:13:43.825280 236126 storage/replica_raft.go:291 [n1,s1,r48/3:/Table/6{6/2-8}] proposing ADD_REPLICA[(n2,s2):4LEARNER]: after=[(n1,s1):3 (n2,s2):4LEARNER] next=5 I190903 20:13:43.827381 236126 storage/store_snapshot.go:995 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] sending LEARNER snapshot d3b348c8 at applied index 59 I190903 20:13:43.827612 236126 storage/store_snapshot.go:1038 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] streamed snapshot to (n2,s2):4LEARNER: kv pairs: 19, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.828840 236605 storage/replica_raftstorage.go:793 [n2,s2,r48/4:{-}] applying LEARNER snapshot [id=d3b348c8 index=59] I190903 20:13:43.831022 236605 storage/replica_raftstorage.go:814 [n2,s2,r48/4:/Table/6{6/2-8}] applied LEARNER snapshot [total=2ms ingestion=4@1ms id=d3b348c8 index=59] I190903 20:13:43.832738 236126 storage/replica_command.go:1430 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] change replicas (add [(n2,s2):4] remove []): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):3, (n2,s2):4LEARNER, next=5, gen=56] I190903 20:13:43.835109 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.838145 236126 storage/replica_raft.go:291 [n1,s1,r48/3:/Table/6{6/2-8}] proposing ADD_REPLICA[(n2,s2):4]: after=[(n1,s1):3 (n2,s2):4] next=5 I190903 20:13:43.838753 236126 storage/replica_command.go:1430 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] change replicas (add [] remove [(n1,s1):3]): existing descriptor r48:/Table/6{6/2-8} [(n1,s1):3, (n2,s2):4, next=5, gen=57] I190903 20:13:43.841333 2227 storage/queue.go:518 [n3,s3,r9/3:/Table/1{3-4}] rate limited in MaybeAdd (raftlog): context canceled I190903 20:13:43.855636 236741 storage/replica_command.go:1430 [n2,replicate,s2,r192/2:/Table/95/1/{4-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r192:/Table/95/1/{4-5} [(n1,s1):1, (n2,s2):2, next=3, gen=163] I190903 20:13:43.857013 236126 storage/replica_raft.go:291 [n1,s1,r48/3:/Table/6{6/2-8}] proposing REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 E190903 20:13:43.857073 236126 storage/replica_raft.go:304 [n1,s1,r48/3:/Table/6{6/2-8}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:13:43.860518 236126 storage/replica_command.go:979 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] could not promote [] to voter, rolling back: change replicas of r48 failed: storage/replica_raft.go:305: [n1,s1,r48/3:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) E190903 20:13:43.860544 236126 storage/queue.go:1027 [n1,replicate,s1,r48/3:/Table/6{6/2-8}] change replicas of r48 failed: storage/replica_raft.go:305: [n1,s1,r48/3:/Table/6{6/2-8}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):3]: after=[(n2,s2):4] next=5 to remove self (leaseholder) I190903 20:13:43.860819 236778 storage/replica_command.go:1430 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r110:/Table/9{3/2-5} [(n1,s1):1, next=2, gen=159] I190903 20:13:43.864454 236741 storage/replica_raft.go:291 [n2,s2,r192/2:/Table/95/1/{4-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n2,s2):2] next=3 I190903 20:13:43.866286 236778 storage/replica_raft.go:291 [n1,s1,r110/1:/Table/9{3/2-5}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:43.867412 236778 storage/store_snapshot.go:995 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] sending LEARNER snapshot b4c3a569 at applied index 19 I190903 20:13:43.867638 236778 storage/store_snapshot.go:1038 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:43.868673 236745 storage/replica_raftstorage.go:793 [n3,s3,r110/2:{-}] applying LEARNER snapshot [id=b4c3a569 index=19] I190903 20:13:43.869939 236745 storage/replica_raftstorage.go:814 [n3,s3,r110/2:/Table/9{3/2-5}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=b4c3a569 index=19] I190903 20:13:43.870787 236778 storage/replica_command.go:1430 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] change replicas (add [(n3,s3):2] remove []): existing descriptor r110:/Table/9{3/2-5} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=160] I190903 20:13:43.873421 236674 storage/store.go:2593 [n1,replicaGC,s1,r192/1:/Table/95/1/{4-5}] removing replica r192/1 I190903 20:13:43.873901 236674 storage/replica_destroy.go:152 [n1,replicaGC,s1,r192/1:/Table/95/1/{4-5}] removed 10 (0+10) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:43.878256 236778 storage/replica_raft.go:291 [n1,s1,r110/1:/Table/9{3/2-5}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:43.878983 236778 storage/replica_command.go:1430 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r110:/Table/9{3/2-5} [(n1,s1):1, (n3,s3):2, next=3, gen=161] I190903 20:13:43.885462 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:13:43.889254 236778 storage/replica_raft.go:291 [n1,s1,r110/1:/Table/9{3/2-5}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 E190903 20:13:43.889316 236778 storage/replica_raft.go:304 [n1,s1,r110/1:/Table/9{3/2-5}] received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) I190903 20:13:43.890116 236778 storage/replica_command.go:979 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] could not promote [] to voter, rolling back: change replicas of r110 failed: storage/replica_raft.go:305: [n1,s1,r110/1:/Table/9{3/2-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:43.890158 236778 storage/queue.go:1027 [n1,replicate,s1,r110/1:/Table/9{3/2-5}] change replicas of r110 failed: storage/replica_raft.go:305: [n1,s1,r110/1:/Table/9{3/2-5}]: received invalid ChangeReplicasTrigger REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 to remove self (leaseholder) E190903 20:13:43.908509 236965 storage/queue.go:1027 [n3,replicate,s3,r193/2:/Table/95/1/{5-6}] no removable replicas from range that needs a removal: [1:0, 2*:25] I190903 20:13:43.920864 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a > 5 I190903 20:13:43.922930 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a < 4 I190903 20:13:43.924924 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 4 I190903 20:13:43.928585 231490 ccl/partitionccl/partition_test.go:216 query: SELECT count(*) FROM "single col list partitioning - DEFAULT" WHERE a = 5 I190903 20:13:43.932550 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=95 (unpartitioned) version=2 mtime=2019-09-03 20:13:43.931274094 +0000 UTC I190903 20:13:43.945114 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 95 1}] I190903 20:13:43.946040 237071 sql/lease.go:923 new lease: 95("unpartitioned") ver=2:1567541937.098990897,0, refcount=0 E190903 20:13:43.981674 237491 storage/queue.go:1027 [n3,replicate,s3,r191/3:/Table/95{-/1}] no removable replicas from range that needs a removal: [1:0, 3*:32] I190903 20:13:43.997241 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=95 (unpartitioned) version=3 mtime=2019-09-03 20:13:43.997045353 +0000 UTC I190903 20:13:44.006290 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 95 2}] I190903 20:13:44.007270 237212 sql/lease.go:923 new lease: 95("unpartitioned") ver=3:1567541937.099990897,0, refcount=0 I190903 20:13:44.032294 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=95 (unpartitioned) version=4 mtime=2019-09-03 20:13:44.02918412 +0000 UTC I190903 20:13:44.040857 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "alter_table", target: 95, info: {TableName:data.public.unpartitioned Statement:ALTER TABLE unpartitioned PARTITION BY NOTHING User:root MutationID:0 CascadeDroppedViews:[]} I190903 20:13:44.044480 238192 storage/replica_command.go:1430 [n2,s2,r192/2:/Table/95/1/{4-5}] change replicas (add [(n1,s1):3LEARNER] remove []): existing descriptor r192:/Table/95/1/{4-5} [(n2,s2):2, next=3, gen=164] I190903 20:13:44.045140 238187 sql/lease.go:923 new lease: 95("unpartitioned") ver=4:1567541937.100990897,0, refcount=0 I190903 20:13:44.048077 934 sql/lease.go:329 [n1,client=127.0.0.1:50666,user=root,scExec] waiting for 1 leases to expire: desc=[{unpartitioned 95 3}] I190903 20:13:44.050668 238177 storage/replica_command.go:1430 [n1,s1,r194/1:/Table/95/{1/6-2}] change replicas (add [(n3,s3):2LEARNER] remove []): existing descriptor r194:/Table/95/{1/6-2} [(n1,s1):1, next=2, gen=163] I190903 20:13:44.051163 238192 storage/replica_raft.go:291 [n2,s2,r192/2:/Table/95/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3LEARNER]: after=[(n2,s2):2 (n1,s1):3LEARNER] next=4 I190903 20:13:44.053093 238192 storage/store_snapshot.go:995 [n2,s2,r192/2:/Table/95/1/{4-5}] sending LEARNER snapshot d92c84ee at applied index 32 I190903 20:13:44.053340 238192 storage/store_snapshot.go:1038 [n2,s2,r192/2:/Table/95/1/{4-5}] streamed snapshot to (n1,s1):3LEARNER: kv pairs: 17, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s W190903 20:13:44.053569 237946 storage/replica_command.go:1917 [n3,s3,r193/2:/Table/95/1/{5-6}] while transferring lease: unable to find store 2 in range r193:/Table/95/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=164] I190903 20:13:44.054080 237946 storage/replica_command.go:1430 [n3,s3,r193/2:/Table/95/1/{5-6}] change replicas (add [] remove [(n1,s1):1]): existing descriptor r193:/Table/95/1/{5-6} [(n1,s1):1, (n3,s3):2, next=3, gen=164] I190903 20:13:44.054724 238661 storage/replica_raftstorage.go:793 [n1,s1,r192/3:{-}] applying LEARNER snapshot [id=d92c84ee index=32] I190903 20:13:44.056949 238177 storage/replica_raft.go:291 [n1,s1,r194/1:/Table/95/{1/6-2}] proposing ADD_REPLICA[(n3,s3):2LEARNER]: after=[(n1,s1):1 (n3,s3):2LEARNER] next=3 I190903 20:13:44.058208 238177 storage/store_snapshot.go:995 [n1,s1,r194/1:/Table/95/{1/6-2}] sending LEARNER snapshot eb0a8a89 at applied index 19 I190903 20:13:44.058421 238177 storage/store_snapshot.go:1038 [n1,s1,r194/1:/Table/95/{1/6-2}] streamed snapshot to (n3,s3):2LEARNER: kv pairs: 9, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s I190903 20:13:44.060692 238587 storage/replica_raftstorage.go:793 [n3,s3,r194/2:{-}] applying LEARNER snapshot [id=eb0a8a89 index=19] I190903 20:13:44.069457 238661 storage/replica_raftstorage.go:814 [n1,s1,r192/3:/Table/95/1/{4-5}] applied LEARNER snapshot [total=15ms ingestion=4@14ms id=d92c84ee index=32] I190903 20:13:44.069972 238192 storage/replica_command.go:1430 [n2,s2,r192/2:/Table/95/1/{4-5}] change replicas (add [(n1,s1):3] remove []): existing descriptor r192:/Table/95/1/{4-5} [(n2,s2):2, (n1,s1):3LEARNER, next=4, gen=165] I190903 20:13:44.072035 237946 storage/replica_raft.go:291 [n3,s3,r193/2:/Table/95/1/{5-6}] proposing REMOVE_REPLICA[(n1,s1):1]: after=[(n3,s3):2] next=3 I190903 20:13:44.072750 238587 storage/replica_raftstorage.go:814 [n3,s3,r194/2:/Table/95/{1/6-2}] applied LEARNER snapshot [total=12ms ingestion=4@12ms id=eb0a8a89 index=19] I190903 20:13:44.075028 238177 storage/replica_command.go:1430 [n1,s1,r194/1:/Table/95/{1/6-2}] change replicas (add [(n3,s3):2] remove []): existing descriptor r194:/Table/95/{1/6-2} [(n1,s1):1, (n3,s3):2LEARNER, next=3, gen=164] I190903 20:13:44.078330 237946 storage/replica_command.go:1430 [n3,s3,r193/2:/Table/95/1/{5-6}] change replicas (add [(n2,s2):3LEARNER] remove []): existing descriptor r193:/Table/95/1/{5-6} [(n3,s3):2, next=3, gen=165] === RUN TestRepartitioning/single_col_range_partitioning/unpartitioned I190903 20:13:44.083548 238192 storage/replica_raft.go:291 [n2,s2,r192/2:/Table/95/1/{4-5}] proposing ADD_REPLICA[(n1,s1):3]: after=[(n2,s2):2 (n1,s1):3] next=4 I190903 20:13:44.108408 238177 storage/replica_raft.go:291 [n1,s1,r194/1:/Table/95/{1/6-2}] proposing ADD_REPLICA[(n3,s3):2]: after=[(n1,s1):1 (n3,s3):2] next=3 I190903 20:13:44.112355 238706 storage/replica_command.go:1430 [n1,s1,r192/3:/Table/95/1/{4-5}] change replicas (add [] remove [(n2,s2):2]): existing descriptor r192:/Table/95/1/{4-5} [(n2,s2):2, (n1,s1):3, next=4, gen=166] E190903 20:13:44.113955 237695 storage/queue.go:1027 [n1,replicate,s1,r192/3:/Table/95/1/{4-5}] no removable replicas from range that needs a removal: [2:0, 3*:37] I190903 20:13:44.115300 237946 storage/replica_raft.go:291 [n3,s3,r193/2:/Table/95/1/{5-6}] proposing ADD_REPLICA[(n2,s2):3LEARNER]: after=[(n3,s3):2 (n2,s2):3LEARNER] next=4 I190903 20:13:44.133888 237946 storage/store_snapshot.go:995 [n3,s3,r193/2:/Table/95/1/{5-6}] sending LEARNER snapshot 0a43b4f8 at applied index 30 I190903 20:13:44.134122 237946 storage/store_snapshot.go:1038 [n3,s3,r193/2:/Table/95/1/{5-6}] streamed snapshot to (n2,s2):3LEARNER: kv pairs: 16, log entries: 0, rate-limit: 8.0 MiB/sec, 0.01s I190903 20:13:44.135322 237666 storage/replica_raftstorage.go:793 [n2,s2,r193/3:{-}] applying LEARNER snapshot [id=0a43b4f8 index=30] I190903 20:13:44.136650 237666 storage/replica_raftstorage.go:814 [n2,s2,r193/3:/Table/95/1/{5-6}] applied LEARNER snapshot [total=1ms ingestion=4@1ms id=0a43b4f8 index=30] I190903 20:13:44.137590 237946 storage/replica_command.go:1430 [n3,s3,r193/2:/Table/95/1/{5-6}] change replicas (add [(n2,s2):3] remove []): existing descriptor r193:/Table/95/1/{5-6} [(n3,s3):2, (n2,s2):3LEARNER, next=4, gen=166] I190903 20:13:44.143806 238706 storage/replica_raft.go:291 [n1,s1,r192/3:/Table/95/1/{4-5}] proposing REMOVE_REPLICA[(n2,s2):2]: after=[(n1,s1):3] next=4 E190903 20:13:44.145304 238980 storage/queue.go:1027 [n3,replicate,s3,r194/2:/Table/95/{1/6-2}] no removable replicas from range that needs a removal: [1:0, 2*:23] I190903 20:13:44.160042 237953 storage/store.go:2593 [n1,replicaGC,s1,r193/1:/Table/95/1/{5-6}] removing replica r193/1 I190903 20:13:44.160393 237953 storage/replica_destroy.go:152 [n1,replicaGC,s1,r193/1:/Table/95/1/{5-6}] removed 9 (0+9) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:44.160444 238327 storage/replica_command.go:609 [n1,merge,s1,r195/1:/Table/95/1{-/4}] initiating a merge of r192:/Table/95/1/{4-5} [(n1,s1):3, next=4, gen=167] into this range (lhs+rhs has (size=0 B+0 B qps=0.00+0.00 --> 0.00qps) below threshold (size=0 B, qps=0.00)) I190903 20:13:44.164718 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root] publish: descID=95 (unpartitioned) version=5 mtime=2019-09-03 20:13:44.081469034 +0000 UTC I190903 20:13:44.166868 239049 storage/store.go:2593 [n2,replicaGC,s2,r192/2:/Table/95/1/{4-5}] removing replica r192/2 I190903 20:13:44.167260 239049 storage/replica_destroy.go:152 [n2,replicaGC,s2,r192/2:/Table/95/1/{4-5}] removed 13 (0+13) keys in 0ms [clear=0ms commit=0ms] I190903 20:13:44.177604 237946 storage/replica_raft.go:291 [n3,s3,r193/2:/Table/95/1/{5-6}] proposing ADD_REPLICA[(n2,s2):3]: after=[(n3,s3):2 (n2,s2):3] next=4 I190903 20:13:44.178839 180 storage/store.go:2593 [n1,s1,r195/1:/Table/95/1{-/4}] removing replica r192/3 E190903 20:13:44.180826 239129 storage/queue.go:1027 [n2,replicate,s2,r193/3:/Table/95/1/{5-6}] no removable replicas from range that needs a removal: [2:0, 3*:34] I190903 20:13:44.187921 934 sql/event_log.go:130 [n1,client=127.0.0.1:50666,user=root] Event: "drop_database", target: 94, info: {DatabaseName:data Statement:DROP DATABASE IF EXISTS data User:root DroppedSchemaObjects:[data.public.unpartitioned]} I190903 20:13:44.195525 45 storage/queue.go:518 [n1,s1] rate limited in MaybeAdd (merge): throttled on async limiting semaphore I190903 20:13:44.203395 934 sql/sqlbase/structured.go:1511 [n1,client=127.0.0.1:50666,user=root,scExec] publish: descID=95 (unpartitioned) version=6 mtime=2019-09-03 20:13:44.203203677 +0000 UTC I190903 20:13:44.210546 239670 storage/replica_command.go:1430 [n1,s1,r195/1:/Table/95/1{-/5}] change replicas (add [(n3,s3):3LEARNER] remove []): existing descriptor r195:/Table/95/1{-/5} [(n1,s1):1, next=3, gen=168] I190903 20:13:44.218706 239670 storage/replica_raft.go:291 [n1,s1,r195/1:/Table/95/1{-/5}] proposing ADD_REPLICA[(n3,s3):3LEARNER]: after=[(n1,s1):1 (n3,s3):3LEARNER] next=4 I190903 20:13:44.220157 239670 storage/store_snapshot.go:995 [n1,s1,r195/1:/Table/95/1{-/5}] sending LEARNER snapshot 91865658 at applied index 19 I190903 20:13:44.220581 239670 storage/store_snapshot.go:1038 [n1,s1,r195/1:/Table/95/1{-/5}] streamed snapshot to (n3,s3):3LEARNER: kv pairs: 29, log entries: 0, rate-limit: 8.0 MiB/sec, 0.00s F190903 20:13:44.222669 238865 storage/replica_raft.go:1486 [n3,s3,r195/2:/Table/95/1{-/4}] snapshot widens existing replica, but no replica exists for subsumed key /Table/95/1/4 goroutine 238865 [running]: github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc00041c301, 0xc00041c300, 0x0, 0x6) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1016 +0xb1 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000004, 0x6bde7e6, 0x17, 0x5ce, 0xc00339ea80, 0x76) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:874 +0x93e github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc00628d8c0, 0xc000000004, 0x2, 0x4038bc6, 0x4b, 0xc003464a58, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc00628d8c0, 0x1, 0xc000000004, 0x4038bc6, 0x4b, 0xc003464a58, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Fatalf(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:180 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).maybeAcquireSnapshotMergeLock(0xc0077a5800, 0x489e1a0, 0xc00628d8c0, 0x1f41627a58568691, 0xcbf673d5b47f9e8c, 0xc005ebc4c0, 0x0, 0x0, 0x0, 0xc003838f00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:1486 +0x294 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked(0xc0077a5800, 0x489e1a0, 0xc00628d8c0, 0x1f41627a58568691, 0xcbf673d5b47f9e8c, 0xc005ebc4c0, 0x0, 0x0, 0x0, 0xc003838f00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:521 +0x1893 github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x489e1a0, 0xc00628d8c0, 0xc0077a5800, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3488 +0x19d github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc00084c700, 0x489e1a0, 0xc00628d8c0, 0xc003838cc8, 0xc006bf1418, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3368 +0x150 github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc00084c700, 0x489e1a0, 0xc00628ce10, 0xc003838c80, 0x1f41627a58568691, 0xcbf673d5b47f9e8c, 0xc005ebc4c0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3430 +0x1a4 github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc00084c700, 0x489e1a0, 0xc00628ce10, 0xc003838c80, 0x7f6d05d411f0, 0xc008422e70, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:871 +0x6d3 github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1(0x489e1a0, 0xc00628ce10, 0xc0019e4510, 0x3fbe40a) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3234 +0x1d9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0019e4510, 0x489e1a0, 0xc00628ce10, 0x3fbe40a, 0x1e, 0xc006bf1c30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc00084c700, 0xc003838c80, 0x7f6d05d411c0, 0xc008422e70, 0xc008422e70, 0x30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3224 +0xe6 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x48fa640, 0xc008422e70, 0xc0018a82d0, 0x489e1a0, 0xc00628cd50, 0x71f47f, 0xc0026bca30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x489e1a0, 0xc00628cd50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc00628cd50, 0xc005390380, 0x32, 0x0, 0x0, 0xc00628cd80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1 [chan receive]: testing.(*T).Run(0xc000177b00, 0x3f999d0, 0x12, 0x40df0a8, 0x7ad001) /usr/local/go/src/testing/testing.go:917 +0x381 testing.runTests.func1(0xc000177a00) /usr/local/go/src/testing/testing.go:1157 +0x78 testing.tRunner(0xc000177a00, 0xc000867e10) /usr/local/go/src/testing/testing.go:865 +0xc0 testing.runTests(0xc0002bc060, 0x6fb55a0, 0x8, 0x8, 0x0) /usr/local/go/src/testing/testing.go:1155 +0x2a9 testing.(*M).Run(0xc000336d00, 0x0) /usr/local/go/src/testing/testing.go:1072 +0x162 github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestMain(0xc000336d00) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/main_test.go:31 +0xda main.main() _testmain.go:54 +0x13e goroutine 20 [syscall]: os/signal.signal_recv(0x0) /usr/local/go/src/runtime/sigqueue.go:139 +0x9c os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:23 +0x22 created by os/signal.init.0 /usr/local/go/src/os/signal/signal_unix.go:29 +0x41 goroutine 21 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.flushDaemon() /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1222 +0xf3 created by github.com/cockroachdb/cockroach/pkg/util/log.init.0 /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:625 +0x121 goroutine 22 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/log.signalFlusher() /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:632 +0x126 created by github.com/cockroachdb/cockroach/pkg/util/log.init.0 /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:626 +0x139 goroutine 60 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc000be6420) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248670, 0xc000a30090, 0xc0003340a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 5 [select]: github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.(*worker).start(0xc000392540) /go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:152 +0xdd created by github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view.init.0 /go/src/github.com/cockroachdb/cockroach/vendor/go.opencensus.io/stats/view/worker.go:29 +0x57 goroutine 59 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc000c3c000) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037c1e0, 0xc000a30090, 0xc00037c1d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 58 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc000c30000) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037c110, 0xc000a30090, 0xc00038adc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 57 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc000a30090, 0xc000af6580) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 10 [chan receive]: testing.(*T).Run(0xc0073ed400, 0xc002a31320, 0x2b, 0xc004a043f0, 0x2) /usr/local/go/src/testing/testing.go:917 +0x381 github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestRepartitioning(0xc000177b00) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/partition_test.go:1291 +0x3a1 testing.tRunner(0xc000177b00, 0x40df0a8) /usr/local/go/src/testing/testing.go:865 +0xc0 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:916 +0x35a goroutine 11 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00020baa0, 0x0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00020ba90) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc00020b980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 12 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6030) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249130, 0xc000a30090, 0xc000249120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 13 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6060) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249150, 0xc000a30090, 0xc000249140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 14 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6090) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249170, 0xc000a30090, 0xc000249160) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 15 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac60c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002491b0, 0xc000a30090, 0xc0002491a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 16 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac60f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002491e0, 0xc000a30090, 0xc0002491d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 66 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6120) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249210, 0xc000a30090, 0xc0002491f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 67 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6150) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249240, 0xc000a30090, 0xc000249220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 68 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6180) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249270, 0xc000a30090, 0xc000249250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 69 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6000) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002492a0, 0xc000a30090, 0xc000249280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 70 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00064c3c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002492d0, 0xc000a30090, 0xc0002492b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 71 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00064c4e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002492f0, 0xc000a30090, 0xc0002492e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 72 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00064c570) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249320, 0xc000a30090, 0xc000249310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 73 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc0007b6090) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249350, 0xc000a30090, 0xc000249340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 74 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac61e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249380, 0xc000a30090, 0xc000249370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 75 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac61b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002493b0, 0xc000a30090, 0xc0002493a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 76 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000ac6210) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249460, 0xc000a30090, 0xc000249450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 77 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0001bde60, 0x489e0e0, 0xc000393480) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 78 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc000072120, 0x489e0e0, 0xc000393480) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 79 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc000be6030) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362330, 0xc000a30090, 0xc000c31710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 80 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc000be60c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362350, 0xc000a30090, 0xc000362340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 81 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc000437d60, 0xc00056c310, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 82 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc000ac6240) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362400, 0xc000a30090, 0xc0003820e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 83 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc000be6120) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362460, 0xc000a30090, 0xc000382120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 84 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c229668, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d418, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc00007d400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc00007d400, 0xc0007e6000, 0x74c230, 0x10) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110068, 0x71f47f, 0xc0002b1ff0, 0xc000cc7ef0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110068, 0x40e2688, 0xc000526200, 0xc000302024, 0x0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000526200, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc000be60f0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362470, 0xc000a30090, 0xc000382180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 85 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc00098e000, 0x4886ea0, 0xc000382140, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc000be6210) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362480, 0xc000a30090, 0xc000c318f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 86 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc0003821a0, 0x3d2f600, 0xc000be61e0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc000be8820, 0x486e6e0, 0xc0003821a0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc000be6150) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362490, 0xc000a30090, 0xc0003821c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 87 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc000be6480) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362500, 0xc000a30090, 0xc000382220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 88 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc000be6450) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362530, 0xc000a30090, 0xc000526280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 89 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc000c12a80, 0x4886ea0, 0xc000382240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc000be64b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362550, 0xc000a30090, 0xc000382280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 90 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc000be64e0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003626d0, 0xc000a30090, 0xc000c31d40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 62 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000a24580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 92 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000526540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 93 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc000ac6270) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362800, 0xc000a30090, 0xc000382560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 64 [select]: net.(*pipe).read(0xc00007d780, 0xc0005d8000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc00007d780, 0xc0005d8000, 0x791, 0x791, 0xc0005d800d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc00538c620, 0xc0005d8000, 0x791, 0x791, 0x2e, 0x482c600, 0xc000811ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0007e7058, 0x482c340, 0xc00538c620, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0007e6e00, 0x7f6d0c22ebd8, 0xc00007d780, 0x5, 0xc00007d780, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0007e6e00, 0x40e5100, 0xc0007e6f38, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0007e6e00, 0xc000c1a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000c24f60, 0xc0002268f8, 0x9, 0x9, 0x6fe9a9, 0xc000986780, 0xc000811e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000c24f60, 0xc0002268f8, 0x9, 0x9, 0x9, 0x0, 0xc00041d8b8, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0002268f8, 0x9, 0x9, 0x482c140, 0xc000c24f60, 0x0, 0xc000000000, 0xc000811ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0002268c0, 0xc004d121a0, 0xc004d121a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000b50000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 94 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00047d620, 0xc0000020bb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00047d610) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc00047d500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 95 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc00064c5a0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003629c0, 0xc000a30090, 0xc000c31f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 96 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc000ac6330) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362a80, 0xc000a30090, 0xc000382f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 97 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc000ac62d0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362aa0, 0xc000a30090, 0xc000362a90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 114 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc000ac6390) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362ad0, 0xc000a30090, 0xc000362ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 115 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c229598, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d518, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc00007d500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc00007d500, 0xc001250301, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110080, 0xc00080ee80, 0xc00080ee88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110080, 0x40e2690, 0xc000526240, 0x48faa00, 0xc0012503f0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000526240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc000318fd0, 0xc00097c750) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc00064c600) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362af0, 0xc000a30090, 0xc000382fa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 116 [semacquire]: sync.runtime_SemacquireMutex(0xc000318fd4, 0xc00097cf00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc000318fd0) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc000318fd0, 0xc00097cf50) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc000724000) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362e00, 0xc000a30090, 0xc000383b80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 65 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000d4b80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000c250e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000b50000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 130 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0000d4c80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000c25140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc000986600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 131 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc000986600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 132 [select]: net.(*pipe).read(0xc00007d700, 0xc0005d8800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc00007d700, 0xc0005d8800, 0x75b, 0x75b, 0xc0005d880d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc00538c640, 0xc0005d8800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc000812a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0007e6958, 0x482c340, 0xc00538c640, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0007e6700, 0x7f6d0c22ebd8, 0xc00007d700, 0x5, 0xc00007d700, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0007e6700, 0x40e5100, 0xc0007e6838, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0007e6700, 0xc000c8a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000c25020, 0xc000226b98, 0x9, 0x9, 0xc001ca2000, 0x7f6d0c288de0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000c25020, 0xc000226b98, 0x9, 0x9, 0x9, 0xbdd055, 0xc004d121cc, 0xc002a72318) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000226b98, 0x9, 0x9, 0x482c140, 0xc000c25020, 0x0, 0x0, 0xc004d121c0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000226b60, 0xc004d121c0, 0xc004d121c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc000986600, 0xc00079e270, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc000986600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc000986600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 117 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc0006964d0, 0x489e1a0, 0xc00064c630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000362fa0, 0xc000a30090, 0xc000362f90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 99 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc00064c6c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000342070, 0xc000a30090, 0xc00031a0c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 118 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00029a380, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0005e34a0, 0x22, 0x0, 0x0, 0xc000363190) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 119 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00029a460, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0005e3500, 0x22, 0x0, 0x0, 0xc0003631b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 120 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc00079e450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b20, 0xc000a30090, 0xc0002e8000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 121 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b60, 0xc000a30090, 0xc000363b30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 122 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6895) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b90, 0xc000a30090, 0xc000363b80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 123 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c20, 0xc000a30090, 0xc000363c10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 124 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6889) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c40, 0xc000a30090, 0xc000363c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 125 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6886) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c60, 0xc000a30090, 0xc000363c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 126 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6888) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c80, 0xc000a30090, 0xc000363c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 127 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6898) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064ca50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064ca50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ca0, 0xc000a30090, 0xc000363c90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 128 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064ca80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064ca80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363cc0, 0xc000a30090, 0xc000363cb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 129 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6887) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ce0, 0xc000a30090, 0xc000363cd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 146 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6875) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cb70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cb70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d00, 0xc000a30090, 0xc000363cf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 147 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cc00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cc00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d20, 0xc000a30090, 0xc000363d10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 148 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6892) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064ccc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064ccc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d40, 0xc000a30090, 0xc000363d30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 149 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6896) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cd20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cd20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d60, 0xc000a30090, 0xc000363d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 150 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cd80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cd80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d80, 0xc000a30090, 0xc000363d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 151 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cde0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cde0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363da0, 0xc000a30090, 0xc000363d90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 152 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064ce40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064ce40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363dc0, 0xc000a30090, 0xc000363db0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 153 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cf00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cf00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363de0, 0xc000a30090, 0xc000363dd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 154 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cf60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cf60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e10, 0xc000a30090, 0xc000363df0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 155 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6878) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064c750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064c750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e30, 0xc000a30090, 0xc000363e20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 156 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e60, 0xc000a30090, 0xc000363e50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 157 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6881) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064d050) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064d050) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e90, 0xc000a30090, 0xc000363e80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 158 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064d830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064d830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ee0, 0xc000a30090, 0xc000363eb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 159 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6890) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064d950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064d950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363f50, 0xc000a30090, 0xc000363ef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 160 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064d9b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064d9b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363f80, 0xc000a30090, 0xc000363f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 161 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6893) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064da10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064da10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363fb0, 0xc000a30090, 0xc000363f90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 162 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064da70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064da70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363fe0, 0xc000a30090, 0xc000363fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 163 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6894) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064db00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064db00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278010, 0xc000a30090, 0xc000363ff0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 164 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6883) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064cfc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064cfc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278040, 0xc000a30090, 0xc000278020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 165 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064dbf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064dbf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278070, 0xc000a30090, 0xc000278050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 166 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064dc50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064dc50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002780a0, 0xc000a30090, 0xc000278080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 167 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064dcb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064dcb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002780c0, 0xc000a30090, 0xc0002780b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 168 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68cb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064de30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064de30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002780f0, 0xc000a30090, 0xc0002780e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 169 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6897) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064db90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064db90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278120, 0xc000a30090, 0xc000278110) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 170 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ae) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064de90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064de90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278150, 0xc000a30090, 0xc000278140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 171 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6891) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064de60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064de60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002781e0, 0xc000a30090, 0xc0002781d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 172 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064dec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064dec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278200, 0xc000a30090, 0xc0002781f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 173 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6899) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064def0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064def0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278220, 0xc000a30090, 0xc000278210) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 174 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064df20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064df20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278240, 0xc000a30090, 0xc000278230) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 175 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68bb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e4b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e4b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278260, 0xc000a30090, 0xc000278250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 176 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e4e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e4e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278280, 0xc000a30090, 0xc000278270) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 177 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ab) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002782a0, 0xc000a30090, 0xc000278290) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 178 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002782c0, 0xc000a30090, 0xc0002782b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 179 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002782e0, 0xc000a30090, 0xc0002782d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 180 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e5a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e5a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278300, 0xc000a30090, 0xc0002782f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 181 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278320, 0xc000a30090, 0xc000278310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 182 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278340, 0xc000a30090, 0xc000278330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 183 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278360, 0xc000a30090, 0xc000278350) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 184 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68bf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278380, 0xc000a30090, 0xc000278370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 185 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68cf) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002783a0, 0xc000a30090, 0xc000278390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 186 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e6c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e6c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002783c0, 0xc000a30090, 0xc0002783b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 187 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68aa) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e6f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e6f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002783e0, 0xc000a30090, 0xc0002783d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 188 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278400, 0xc000a30090, 0xc0002783f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 189 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278420, 0xc000a30090, 0xc000278410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 190 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278440, 0xc000a30090, 0xc000278430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 191 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e7b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e7b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278460, 0xc000a30090, 0xc000278450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 192 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68af) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e7e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e7e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278480, 0xc000a30090, 0xc000278470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 193 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064df80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064df80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002784b0, 0xc000a30090, 0xc000278490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 194 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002784d0, 0xc000a30090, 0xc0002784c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 195 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00064dfb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00064dfb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278500, 0xc000a30090, 0xc0002784f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 196 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e8a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e8a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278530, 0xc000a30090, 0xc000278520) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 197 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68bd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e8d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e8d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278560, 0xc000a30090, 0xc000278550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 198 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68cc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278590, 0xc000a30090, 0xc000278580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 199 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002785c0, 0xc000a30090, 0xc0002785a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 200 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002785f0, 0xc000a30090, 0xc0002785d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 201 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278620, 0xc000a30090, 0xc000278600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 202 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002786b0, 0xc000a30090, 0xc000278630) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 203 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ad) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ea20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ea20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002786d0, 0xc000a30090, 0xc0002786c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 204 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ea50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ea50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002786f0, 0xc000a30090, 0xc0002786e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 205 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68a7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ea80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ea80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278710, 0xc000a30090, 0xc000278700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 206 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ba) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079eae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079eae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278730, 0xc000a30090, 0xc000278720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 207 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079eb10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079eb10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278750, 0xc000a30090, 0xc000278740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 208 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079e840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079e840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278770, 0xc000a30090, 0xc000278760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 209 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079eb70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079eb70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278790, 0xc000a30090, 0xc000278780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 210 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68dc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079eba0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079eba0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002787b0, 0xc000a30090, 0xc0002787a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 211 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0007240c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0007240c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002787d0, 0xc000a30090, 0xc0002787c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 212 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ca) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000930000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000930000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002787f0, 0xc000a30090, 0xc0002787e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 213 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68cd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000ac63c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000ac63c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278810, 0xc000a30090, 0xc000278800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 214 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278830, 0xc000a30090, 0xc000278820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 215 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079eb40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079eb40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278850, 0xc000a30090, 0xc000278840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 216 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68b7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ec30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ec30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278870, 0xc000a30090, 0xc000278860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 217 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ec60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ec60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278890, 0xc000a30090, 0xc000278880) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 218 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68da) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ec00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ec00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002788b0, 0xc000a30090, 0xc0002788a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 219 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68bc) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ecc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ecc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002788d0, 0xc000a30090, 0xc0002788c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 220 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ec90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ec90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002788f0, 0xc000a30090, 0xc0002788e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 221 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ecf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ecf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278910, 0xc000a30090, 0xc000278900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 222 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc00079ed20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00079ed20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278930, 0xc000a30090, 0xc000278920) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 223 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ce) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278950, 0xc000a30090, 0xc000278940) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 224 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278970, 0xc000a30090, 0xc000278960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 225 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009680f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009680f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002789a0, 0xc000a30090, 0xc000278990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 226 [runnable]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6868) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002789d0, 0xc000a30090, 0xc0002789b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 227 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278a00, 0xc000a30090, 0xc0002789e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 228 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68dd) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a8000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a8000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278a30, 0xc000a30090, 0xc000278a10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 229 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68db) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278a60, 0xc000a30090, 0xc000278a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 230 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68df) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278a80, 0xc000a30090, 0xc000278a70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 231 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000930060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000930060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ab0, 0xc000a30090, 0xc000278aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 232 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000930090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000930090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ae0, 0xc000a30090, 0xc000278ad0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 233 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009300c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009300c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278b10, 0xc000a30090, 0xc000278b00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 234 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6873) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000930030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000930030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ba0, 0xc000a30090, 0xc000278b90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 235 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6872) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a8090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a8090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278bc0, 0xc000a30090, 0xc000278bb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 236 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a8030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a8030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278be0, 0xc000a30090, 0xc000278bd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 237 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6874) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a80c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a80c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278c00, 0xc000a30090, 0xc000278bf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 238 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6870) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a80f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a80f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278c20, 0xc000a30090, 0xc000278c10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 239 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68c3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278c40, 0xc000a30090, 0xc000278c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 240 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278c60, 0xc000a30090, 0xc000278c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 241 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968270) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278c80, 0xc000a30090, 0xc000278c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 242 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6885) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009682a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009682a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ca0, 0xc000a30090, 0xc000278c90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 243 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009682d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009682d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278cc0, 0xc000a30090, 0xc000278cb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 244 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6880) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ce0, 0xc000a30090, 0xc000278cd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 245 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68be) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278d00, 0xc000a30090, 0xc000278cf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 246 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68d5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009681b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009681b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278d20, 0xc000a30090, 0xc000278d10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 247 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x689a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009683c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009683c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278d40, 0xc000a30090, 0xc000278d30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 248 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6871) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278d60, 0xc000a30090, 0xc000278d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 249 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6884) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278d80, 0xc000a30090, 0xc000278d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 250 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x686c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278da0, 0xc000a30090, 0xc000278d90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 251 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6879) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009684b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009684b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278dc0, 0xc000a30090, 0xc000278db0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 252 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68de) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278de0, 0xc000a30090, 0xc000278dd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 253 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009684e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009684e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278e00, 0xc000a30090, 0xc000278df0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 254 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278e20, 0xc000a30090, 0xc000278e10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 255 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278e40, 0xc000a30090, 0xc000278e30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 256 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68e0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009685a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009685a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278e60, 0xc000a30090, 0xc000278e50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 257 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6882) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009685d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009685d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278e80, 0xc000a30090, 0xc000278e70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 258 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6876) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278eb0, 0xc000a30090, 0xc000278ea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 259 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x688f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278ee0, 0xc000a30090, 0xc000278ec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 260 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6877) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278f10, 0xc000a30090, 0xc000278ef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 261 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x6869) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278f40, 0xc000a30090, 0xc000278f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 262 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc000968510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000968510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278f60, 0xc000a30090, 0xc000278f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 263 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x68ac) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009300f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009300f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278f90, 0xc000a30090, 0xc000278f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 264 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0005272d0, 0x687a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0005272c0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc00007df80, 0x489e1a0, 0xc0009a8120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0009a8120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278fc0, 0xc000a30090, 0xc000278fb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 265 [semacquire]: sync.runtime_Semacquire(0xc00007dff8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc00007dff0) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279030, 0xc000a30090, 0xc000279020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 266 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc000a15500, 0x489e1a0, 0xc0009686c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279050, 0xc000a30090, 0xc000279040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 267 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc000a15500, 0x489e1a0, 0xc000968720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279080, 0xc000a30090, 0xc000279060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 268 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc0009686f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002790d0, 0xc000a30090, 0xc0002e8020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 269 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000930150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002790e0, 0xc000a30090, 0xc000befef0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 270 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000968780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002790f0, 0xc000a30090, 0xc000beff40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 271 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc000968990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279100, 0xc000a30090, 0xc000beff90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1522 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc001d745a0, 0xc0003c7610, 0xf, 0xc001804120, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001a1e3f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68410, 0xc000937c20, 0xc0019a00f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 273 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc000ac6420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279140, 0xc000a30090, 0xc000279130) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 290 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000527210, 0xa1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000527200) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0006964d0, 0x489e1a0, 0xc0005adbf0, 0xc000bebe60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc0005adbf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e1a0, 0xc0005adbf0, 0xc0005b8fc0, 0x25, 0x0, 0x0, 0xc0002e8060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 291 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc000ac6660) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279150, 0xc000a30090, 0xc0002e8080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 292 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc000ac66c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279160, 0xc000a30090, 0xc0002e80a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 293 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc000969b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279170, 0xc000a30090, 0xc0002e80c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 36 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac64b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002600c0, 0xc000a30090, 0xc0003222e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 106 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000886840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1523 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000a24840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1810 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087eed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e80, 0xc000937c20, 0xc000335840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 10236 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00368ad80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 37 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac64e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002600d0, 0xc000a30090, 0xc000322300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 38 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac6510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260100, 0xc000a30090, 0xc000322320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 39 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac6540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260110, 0xc000a30090, 0xc000322360) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 40 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac6570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260120, 0xc000a30090, 0xc000322380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 41 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac65a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260130, 0xc000a30090, 0xc0003223a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 42 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac65d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260140, 0xc000a30090, 0xc0003223c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 43 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac6600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260150, 0xc000a30090, 0xc0003223e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 44 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc000ac6630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260170, 0xc000a30090, 0xc000322400) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 45 [semacquire]: sync.runtime_SemacquireMutex(0x726cb6c, 0xc91b51600) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0x726cb68) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000001, 0x6bde990, 0x10, 0x206, 0xc00336d9a0, 0x4f) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:857 +0x206 github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc000ac6480, 0xc000000001, 0x2, 0x3fa5231, 0x16, 0xc004d2cc80, 0x2, 0x2) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc000ac6480, 0x1, 0xc000000001, 0x3fa5231, 0x16, 0xc004d2cc80, 0x2, 0x2) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Infof(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:95 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).Async(0xc000158c80, 0x489e1a0, 0xc000ac6480, 0xc00564bfa0, 0x10, 0xc000437d00, 0xc0009bc450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:518 +0x273 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).MaybeAddAsync(0xc000158c80, 0x489e1a0, 0xc000ac6480, 0x4906de0, 0xc005d42800, 0x15c106cd566883f1, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:526 +0xac github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc000be9930, 0x489e1a0, 0xc000ac6480, 0xc000a30090, 0x24fc62c4, 0xed500c077, 0x0, 0xc005d42800, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:231 +0x326 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc005d42800, 0xc0005ad601) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc0005ad6e0, 0xc005b32600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc000ac6480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260180, 0xc000a30090, 0xc000322420) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 294 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc0007240f0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279520, 0xc000a30090, 0xc0002e8340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 295 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc000969c50) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279580, 0xc000a30090, 0xc0002e83a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 296 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc000437d60, 0xc00017fea0, 0xc0002795e0, 0x40e5438, 0xc00041d140, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc000969ce0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002795f0, 0xc000a30090, 0xc0008f26c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 297 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc000969d10) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279600, 0xc000a30090, 0xc0002e8440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 298 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc000969d70) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002796f0, 0xc000a30090, 0xc0008f2780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 299 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc000969e30) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279720, 0xc000a30090, 0xc000279700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 300 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc0009a8150) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279730, 0xc000a30090, 0xc0002e8540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 301 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc000969d40) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279780, 0xc000a30090, 0xc000279760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 302 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc000969dd0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279790, 0xc000a30090, 0xc0002e8560) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 303 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc000c3c3f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002797a0, 0xc000a30090, 0xc0002e8580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 304 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc0008f2a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279930, 0xc000a30090, 0xc000f54140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1377 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001d642c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 865 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00262b620, 0xc000000000) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00262b610) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc00262b500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 1376 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc001af41a0, 0x489e1a0, 0xc001a1e240, 0xc00193c780, 0x48f9ec0, 0xc0019b4100, 0xc000937c20, 0xc0019d64b4, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:366 +0x344 github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x489e1a0, 0xc001a1e240) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:125 +0x3c2 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68260, 0xc000937c20, 0xc001d64080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 347 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc000930180) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037cb70, 0xc000a30090, 0xc000341080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2018 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606a50) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311540, 0xc0019e4510, 0xc000311530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 348 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc000c3d4a0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037cb80, 0xc000a30090, 0xc0009a93e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 349 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc000c3d470) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037cb90, 0xc000a30090, 0xc0003410a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 350 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc000c3d4d0) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037cba0, 0xc000a30090, 0xc0009a9410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 351 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc000c3d500) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037cbb0, 0xc000a30090, 0xc0003410e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1374 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc0008f2030) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d681b0, 0xc000937c20, 0xc001d681a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 320 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000f90ac0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000f4f4a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0011f0300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 318 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c2293f8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000337c18, 0x72, 0x6100, 0x6115, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000337c00, 0xc00423ca80, 0x6115, 0x6115, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000337c00, 0xc00423ca80, 0x6115, 0x6115, 0x203001, 0x0, 0x6108) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000010008, 0xc00423ca80, 0x6115, 0x6115, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00449c160, 0xc00423ca80, 0x6115, 0x6115, 0x42, 0x482c600, 0xc0012a3a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0000e0958, 0x482c340, 0xc00449c160, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0000e0700, 0x48317e0, 0xc000010008, 0x5, 0xc000010008, 0xc002c60030) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0000e0700, 0x40e5100, 0xc0000e0838, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0000e0700, 0xc00119e000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000a95b60, 0xc000fce818, 0x9, 0x9, 0xc005242418, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000a95b60, 0xc000fce818, 0x9, 0x9, 0x9, 0xc00156b230, 0x485e320, 0xc003de74c0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000fce818, 0x9, 0x9, 0x482c140, 0xc000a95b60, 0x0, 0xc000000000, 0xc0017d2000, 0x2c60000) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000fce7e0, 0xc002c60000, 0xc002c60000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000b51340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 319 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000f90b00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000a95c20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000b51340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1640 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c1d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf78c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf78c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003635f0, 0xc000937c20, 0xc000363590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 321 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0011f0300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1565 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc001647c20, 0xc0019d75b0, 0xf, 0xc00041c7e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001bf64b0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003437b0, 0xc000937c20, 0xc000f87d60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 356 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0014e0900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 357 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000c22000, 0xc0009f8c60, 0xc00010d850, 0xf, 0xc0008856e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000a3c210) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248670, 0xc000a30090, 0xc000f55360) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2134 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0018ca780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 317 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000b51340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2990 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002d8d880, 0xc001f66120, 0x489e1a0, 0xc0035051d0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1641 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c99) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf78f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf78f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363660, 0xc000937c20, 0xc000363610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7593 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0033ec800, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000fd1b00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001ece180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 238963 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc3, 0x490bae0, 0xc0078b1620, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc3, 0xc003e14c60, 0xc00176f738, 0xc0078bb560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc006692540, 0x33, 0x0, 0x0, 0xc005236ab0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10235 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003a0bc80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002a59b60, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00368ad80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 864 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc0016069f0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000261c10, 0xc0019e4510, 0xc000261c00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2386 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc001582f00) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037d390, 0xc0019e4510, 0xc001582a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2707 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0012348c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2123 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001bdce50, 0x9d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001bdce40) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007844d0, 0x489e1a0, 0xc0015ef950, 0xc002e21a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0015ef950) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e1a0, 0xc0015ef950, 0xc0005e0440, 0x1d, 0x0, 0x0, 0xc0003cd3a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2986 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002c86380, 0xc001a8dd40, 0x489e1a0, 0xc003504b40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1629 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc00116c7e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57130, 0xc000937c20, 0xc000383540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1532 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001d65340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1869 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001892180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2966 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001ce2000, 0xc000d55e60, 0x489e1a0, 0xc003197890) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2709 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018af400, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001b06ae0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0012348c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1533 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc0016746c0, 0xc0019d75b0, 0xf, 0xc001805080, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001a1ffb0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69320, 0xc000937c20, 0xc0019a1ea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3106 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0002a8960, 0x48fa580, 0xc001843080, 0x485c120, 0xc0002a8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc002724480, 0x0, 0xc0019b6fc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc002724480, 0xc00279ce20, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc002724480, 0xc00279ce20, 0x40e1ba8, 0x489e1a0, 0xc0032c1f20) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc00168b200, 0xc002700f00, 0xc000c58270, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc00168b200, 0xc002700f00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000331840, 0xc000c12a80, 0x48fef60, 0xc00168b200, 0xc002700f00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2387 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc001582f30) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037d430, 0xc0019e4510, 0xc00030ffa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2102 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000527210, 0xa0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000527200) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0006964d0, 0x489e1a0, 0xc0015fc390, 0xc002c926c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0015fc390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e1a0, 0xc0015fc390, 0xc00141e080, 0x1d, 0x0, 0x0, 0xc000247aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1906 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3718, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001630398, 0x72, 0x4900, 0x49ae, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001630380, 0xc002688a80, 0x49ae, 0x49ae, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001630380, 0xc002688a80, 0x49ae, 0x49ae, 0x203000, 0x0, 0x49a1) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0000101f0, 0xc002688a80, 0x49ae, 0x49ae, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc003906060, 0xc002688a80, 0x49ae, 0x49ae, 0x76, 0x482c600, 0xc0011eca78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0015a73d8, 0x482c340, 0xc003906060, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0015a7180, 0x48317e0, 0xc0000101f0, 0x5, 0xc0000101f0, 0xc006eb2f30) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0015a7180, 0x40e5100, 0xc0015a72b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0015a7180, 0xc00281a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0018949c0, 0xc0003e1c38, 0x9, 0x9, 0xc0068de238, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0018949c0, 0xc0003e1c38, 0x9, 0x9, 0x9, 0xc003d53170, 0x485e320, 0xc003acb080) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003e1c38, 0x9, 0x9, 0x482c140, 0xc0018949c0, 0x0, 0xc000000000, 0xc007c8c500, 0x6eb2f00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e1c00, 0xc006eb2f00, 0xc006eb2f00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00293e000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1493 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000be4600, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001c7e6c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0001ba000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2000 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b9920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56620, 0xc0019e4510, 0xc00039ed20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2025 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606a80) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311670, 0xc0019e4510, 0xc000311650) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1642 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c87) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003636e0, 0xc000937c20, 0xc000363680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1566 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000887600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1555 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000366a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1534 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000a24b00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1907 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001d8ddc0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001894ae0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00293e000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1639 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c83) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7890) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363580, 0xc000937c20, 0xc000363540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 450 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c229328, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00028a518, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00028a500, 0xc0005d9800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00028a500, 0xc0005d9800, 0x75b, 0x75b, 0xc000ff1938, 0x7f6d0c1ddd01, 0x294a5294a5) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00030c000, 0xc0005d9800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0003ef380, 0xc0005d9800, 0x75b, 0x75b, 0x8, 0x852ca945b4db, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc000355a60, 0xc0005d9800, 0x75b, 0x75b, 0x203001, 0x140, 0xc000ff1a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0004973d8, 0x482c340, 0xc000355a60, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000497180, 0x7f6d0700c008, 0xc0003ef380, 0x5, 0xc0003ef380, 0x0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000497180, 0x40e5100, 0xc0004972b8, 0xc00156b6e0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000497180, 0xc001186000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc000a95a40, 0xc000fce738, 0x9, 0x9, 0xc0062923c8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc000a95a40, 0xc000fce738, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000fce738, 0x9, 0x9, 0x482c140, 0xc000a95a40, 0x0, 0x6300000000, 0xc0017d2200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000fce700, 0xc000a009c0, 0xc000a009c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0011f0300, 0xc000a01140, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc0011f0300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc0011f0300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2989 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00340bae0, 0xc003336ff0, 0x5, 0x5, 0x94, 0xc00307e5f8, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00340bae0, 0xc003336ff0, 0x5, 0x5, 0xc11b10, 0xc001998240, 0xc0036b5858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0031979b0, 0xc003336ff0, 0x5, 0x5, 0xc0073235e0, 0xc0036b5880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0031979b0, 0xc003336ff0, 0x5, 0x5, 0x5, 0xc001b2f080, 0xc0036b5948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f1300, 0xc003336ff0, 0x5, 0x5, 0x40, 0x40, 0xc001c80000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003336fe0, 0x7fffffff, 0xc0e5c1, 0xc000885140, 0x58, 0x58, 0xc003fe33e0, 0xc000249b90) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003336fe0, 0xc0010f1300, 0x0, 0x0, 0x7fffffff, 0xc0036b5b98, 0x4886820, 0x72b56a8, 0x0, 0xc0078656a5, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003336fe0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f1300, 0x0, 0x0, 0x3de9580, 0xc006c34030, 0x7fffffff, 0xc0036b5b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001569480, 0x3de9580, 0xc006c34030, 0xc0036b5b98, 0x0, 0xc006c34030) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001569480, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d55e60, 0xc0036b5ce0, 0xc0036b5cd0, 0x72b56a8, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d55e60, 0x3de9580, 0xc006c34030, 0xc0073235a0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0019b5aa0, 0xc0073235a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc0031978f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8a10, 0xc0019e4510, 0xc001a13ba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1373 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc001a1e150) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68190, 0xc000937c20, 0xc001d68180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1372 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc001a1e120) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68170, 0xc000937c20, 0xc001d2e180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2995 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0037dfe50, 0xc003213310, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0037dfe50, 0xc003213310, 0x5, 0x5, 0xc11b10, 0xc001ce5be0, 0xc003359af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00321ff50, 0xc003213310, 0x5, 0x5, 0x0, 0xc003359b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00321ff50, 0xc003213310, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001633600, 0xc003213310, 0x5, 0x5, 0x0, 0x100000001, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003213300, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003213300, 0xc001633600, 0x0, 0x0, 0x7fffffff, 0xc003359e30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003213300, 0x7f6d0c1990d0, 0xc000292db0, 0xc001633600, 0x0, 0x0, 0x3de86e0, 0xc004caa520, 0x7fffffff, 0xc003359e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001d4ecc0, 0x3de86e0, 0xc004caa520, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001dbd7e0, 0x489e1a0, 0xc0023f8030, 0xc00559f000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001dbd7e0, 0xc0002a8960, 0x489e1a0, 0xc0023f8030, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0023f8030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001dbd800, 0xc000a30090, 0xc0023f8000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2987 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0017a7e00, 0xc001a13950, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0017a7e00, 0xc001a13950, 0x5, 0x5, 0xc11b10, 0xc001ad3ae0, 0xc00397d978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003504c60, 0xc001a13950, 0x5, 0x5, 0x74cff0, 0xc00397d9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003504c60, 0xc001a13950, 0x5, 0x5, 0x5, 0xf, 0xc0011f5620, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0037f6300, 0xc001a13950, 0x5, 0x5, 0x83e500, 0xc0038bc980, 0xc00397da78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001a13940, 0x7fffffff, 0x3c06120, 0xc0011f5620, 0xc00397da0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001a13940, 0xc0037f6300, 0x0, 0x0, 0x7fffffff, 0xc00397dcb8, 0x4886820, 0x72b56a8, 0x0, 0xc007c86500, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001a13940, 0x7f6d0c1990d0, 0xc000292db0, 0xc0037f6300, 0x0, 0x0, 0x3de8800, 0xc006460440, 0x7fffffff, 0xc00397dcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001c6b180, 0x3de8800, 0xc006460440, 0xc00397dcb8, 0xc006144280, 0xc006460440) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001c6b180, 0xc000092f70, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001a8dd40, 0xc00397de00, 0xc00397ddf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001a8dd40, 0x3de8800, 0xc006460440, 0x2c5e8e0, 0xc00084c700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0015a8950, 0x489e1a0, 0xc003504d20, 0xc004c98540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0015a8950, 0xc001c6d780, 0xc0018a82d0, 0x489e1a0, 0xc003504d20, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003504d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8970, 0xc0019e4510, 0xc003504c90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1582 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc001bf7650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363190, 0xc000937c20, 0xc0003b23e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 862 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc0019e4510, 0xc00047ba20) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 2782 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0018a82d0, 0x48fa580, 0xc0015a8890, 0x485c120, 0xc0018a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc001810540, 0x0, 0xc0026381e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc001810540, 0xc001a13620, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc001810540, 0xc001a13620, 0x40e1ba8, 0x489e1a0, 0xc0035044e0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc0022d5200, 0xc0037f6200, 0xc0009a98f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc0022d5200, 0xc0037f6200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000347db0, 0xc00262b200, 0x48fef60, 0xc0022d5200, 0xc0037f6200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1505 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228f18, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001580518, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001580500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001580500, 0xc001d66101, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc001d66458, 0xc001417e68, 0xc001417e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc001d66458, 0x40e2110, 0xc000937c20, 0x489e1a0, 0xc000c3d320) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc001d66100, 0x489e1a0, 0xc000c3d320, 0xc000937c20, 0x4886160, 0xc001d66458, 0xc000c3d350, 0xc000c3d320, 0xc00167a768) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc000c3d320) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b7410, 0xc000937c20, 0xc000be6e70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1515 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00166f800) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1580 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000301880, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e120, 0xc00010c010, 0xc001420db0, 0x22, 0x0, 0x0, 0xc000362fb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1581 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc000301b20, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e120, 0xc00010c010, 0xc001420e10, 0x22, 0x0, 0x0, 0xc000362fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1634 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003633a0, 0xc000937c20, 0xc000363360) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1905 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001ce2700, 0xc001c73680, 0x489e1a0, 0xc00195f2f0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2985 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0018a82d0, 0x1, 0xc001e8de00, 0xc001c6d780, 0x48fa520, 0xc0015a8950, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003504a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a88e0, 0xc0019e4510, 0xc003504a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1995 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b9710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d565d0, 0xc0019e4510, 0xc00039ec60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1207 [select]: database/sql.(*DB).connectionOpener(0xc0017075c0, 0x489e0e0, 0xc000c41a80) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 1204 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c2294c8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d618, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc00007d600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc00007d600, 0xc001dcc001, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000110090, 0xc001d17e68, 0xc001d17e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000110090, 0x40e2110, 0xc000a30090, 0x489e1a0, 0xc0009a8420) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc0016687a8, 0x489e1a0, 0xc0009a8420, 0xc000a30090, 0x4886160, 0xc000110090, 0xc0009a8450, 0xc0009a8420, 0xc000ef9768) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc0009a8420) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5680, 0xc000a30090, 0xc000a3c660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1208 [select]: database/sql.(*DB).connectionResetter(0xc0017075c0, 0x489e0e0, 0xc000c41a80) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 1996 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b9740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d565e0, 0xc0019e4510, 0xc00039eca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1202 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc000a3cb10) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5620, 0xc000a30090, 0xc00030e460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2012 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0015f3240, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0026cbf80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001de0180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2010 [select]: net.(*pipe).read(0xc000f7a600, 0xc001ccf000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc000f7a600, 0xc001ccf000, 0x791, 0x791, 0xc001ccf00d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc005abad40, 0xc001ccf000, 0x791, 0x791, 0x2e, 0x482c600, 0xc0019a4ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001c5a958, 0x482c340, 0xc005abad40, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001c5a700, 0x7f6d0c22ebd8, 0xc000f7a600, 0x5, 0xc000f7a600, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001c5a700, 0x40e5100, 0xc001c5a838, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001c5a700, 0xc002104000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0026cbda0, 0xc000fce9d8, 0x9, 0x9, 0x6fe9a9, 0xc00168aa80, 0xc0019a4e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0026cbda0, 0xc000fce9d8, 0x9, 0x9, 0x9, 0x0, 0xc0015e9138, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000fce9d8, 0x9, 0x9, 0x482c140, 0xc0026cbda0, 0x0, 0xc000000000, 0xc0019a4ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000fce9a0, 0xc005e358a0, 0xc005e358a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00293e1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1618 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228a38, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001bf4118, 0x72, 0x2700, 0x2739, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001bf4100, 0xc001056000, 0x2739, 0x2739, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001bf4100, 0xc001056000, 0x2739, 0x2739, 0xc0016ff938, 0x2701, 0xc001d0a500) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00030c060, 0xc001056000, 0x2739, 0x2739, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc001a217a0, 0xc001056000, 0x2739, 0x2739, 0xc00105600d, 0x5a, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc005827fe0, 0xc001056000, 0x2739, 0x2739, 0x77, 0x482c600, 0xc0016ffa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001d0a5d8, 0x482c340, 0xc005827fe0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001d0a380, 0x7f6d0700c008, 0xc001a217a0, 0x5, 0xc001a217a0, 0x5a) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001d0a380, 0x40e5100, 0xc001d0a4b8, 0x49) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001d0a380, 0xc0014f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001bd4ba0, 0xc0003758b8, 0x9, 0x9, 0xc0019581c8, 0xc0032d4d20, 0x51) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001bd4ba0, 0xc0003758b8, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003758b8, 0x9, 0x9, 0x482c140, 0xc001bd4ba0, 0x0, 0x4900000000, 0xc00244ca00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000375880, 0xc00195eb40, 0xc00195eb40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0009ed080, 0xc001c95dd0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc0009ed080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc0009ed080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2087 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0002a8960, 0x48fa580, 0xc001d68140, 0x485c120, 0xc0002a8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001c44240, 0x0, 0xc001bd4b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001c44240, 0xc000413080, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001c44240, 0xc000413080, 0x40e1ba8, 0x489e1a0, 0xc0008f24e0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc00244ca00, 0xc000c58270, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc00244ca00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001d07520, 0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc00244ca00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2142 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018cb1c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002c85860, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001234700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1481 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0015f2980) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2583 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0015a6e00, 0xc003324240, 0x489e1a0, 0xc001a36ab0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2008 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001ea0000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2833 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000c22000, 0xc003144420, 0xc0003aa520, 0xf, 0xc000a9dda0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001a66ae0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016effe0, 0xc000a30090, 0xc0031eeaa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7698 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0036d5500, 0xc002f8a000, 0x489e1a0, 0xc001583080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2013 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001de0180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2135 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc002c84a20, 0xc000448ed0, 0xf, 0xc002974a20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001d8ab10) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016ee710, 0xc0019e4510, 0xc000345810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2014 [select]: net.(*pipe).read(0xc000f7a580, 0xc001ccf800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc000f7a580, 0xc001ccf800, 0x75b, 0x75b, 0xc001ccf80d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc005abad60, 0xc001ccf800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc0011eda50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0022d2258, 0x482c340, 0xc005abad60, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0022d2000, 0x7f6d0c22ebd8, 0xc000f7a580, 0x5, 0xc000f7a580, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0022d2000, 0x40e5100, 0xc0022d2138, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0022d2000, 0xc00211c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0026cbe60, 0xc000fceab8, 0x9, 0x9, 0xc000208380, 0x7f6d0c274008, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0026cbe60, 0xc000fceab8, 0x9, 0x9, 0x9, 0xbdd055, 0xc005e358cc, 0xc000318a28) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000fceab8, 0x9, 0x9, 0x482c140, 0xc0026cbe60, 0x0, 0x0, 0xc005e358c0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000fcea80, 0xc005e358c0, 0xc005e358c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001de0180, 0xc00087eff0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc001de0180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc001de0180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2884 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0002a8960, 0x2, 0xc0031443c0, 0xc001db86c0, 0x48fa520, 0xc0019b51c0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001b2b200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b4f40, 0xc000a30090, 0xc001b2afc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2011 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0015f3140, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0026cbf20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00293e1c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 235507 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc0073f7aa0, 0x0, 0x1, 0x0, 0x3de2740) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc1, 0xc000a2c120, 0xc0077a5800, 0x15c106cd28c5f30b) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc007086200, 0x33, 0x0, 0x0, 0xc005f73890) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2708 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2ae8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001bf4498, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001bf4480, 0xc00285f000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001bf4480, 0xc00285f000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001250088, 0xc00285f000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00518f7a0, 0xc00285f000, 0x791, 0x791, 0x1b0, 0x482c600, 0xc001a4da78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001a0b058, 0x482c340, 0xc00518f7a0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001a0ae00, 0x48317e0, 0xc001250088, 0x5, 0xc001250088, 0xc0060ff950) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001a0ae00, 0x40e5100, 0xc001a0af38, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001a0ae00, 0xc001622000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001b06960, 0xc00174c578, 0x9, 0x9, 0xc0063ead78, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001b06960, 0xc00174c578, 0x9, 0x9, 0x9, 0xc0060ff590, 0x485e320, 0xc005071d00) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00174c578, 0x9, 0x9, 0x482c140, 0xc001b06960, 0x0, 0xc000000000, 0xc0073ecc00, 0x60ff900) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00174c540, 0xc0060ff920, 0xc0060ff920, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0012348c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 863 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc00158cf90) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000261b30, 0xc0019e4510, 0xc00000fe40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1829 [select]: database/sql.(*DB).connectionResetter(0xc001c44180, 0x489e0e0, 0xc000f906c0) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 1886 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0002a8960, 0x2, 0xc001d00c60, 0xc000394e80, 0x48fa520, 0xc0003426c0, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001d54600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57010, 0xc000a30090, 0xc001d6d7a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1281 [select]: github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler.func1(0xc000937c20, 0xc001a90b00) /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:175 +0x14a created by github.com/cockroachdb/cockroach/pkg/security.(*CertificateManager).RegisterSignalHandler /go/src/github.com/cockroachdb/cockroach/pkg/security/certificate_manager.go:172 +0x49 goroutine 1338 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c270) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8340, 0xc000937c20, 0xc0015a8330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2783 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0017a7d10, 0xc001a13610, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0017a7d10, 0xc001a13610, 0x5, 0x5, 0xc11b10, 0xc001ad39c0, 0xc0036bfaf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003504480, 0xc001a13610, 0x5, 0x5, 0x0, 0xc0036bfb20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003504480, 0xc001a13610, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0037f6200, 0xc001a13610, 0x5, 0x5, 0x0, 0x300000003, 0x3) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001a13600, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001a13600, 0xc0037f6200, 0x0, 0x0, 0x7fffffff, 0xc0036bfe30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001a13600, 0x7f6d0c1990d0, 0xc000292db0, 0xc0037f6200, 0x0, 0x0, 0x3de86e0, 0xc0048d3a00, 0x7fffffff, 0xc0036bfe30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001810540, 0x3de86e0, 0xc0048d3a00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0015a8890, 0x489e1a0, 0xc003504540, 0xc006bfe5f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0015a8890, 0xc0018a82d0, 0x489e1a0, 0xc003504540, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003504540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a88b0, 0xc0019e4510, 0xc003504510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2988 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0017a7f90, 0xc001a13e90, 0x5, 0x5, 0x95, 0xc006053dc0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0017a7f90, 0xc001a13e90, 0x5, 0x5, 0xc11b10, 0xc001ad3b40, 0xc002c83858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0035052f0, 0xc001a13e90, 0x5, 0x5, 0xc00682a2a0, 0xc002c83880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0035052f0, 0xc001a13e90, 0x5, 0x5, 0x5, 0xc00166f680, 0xc002c83948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0037f6400, 0xc001a13e90, 0x5, 0x5, 0xc0032cfe80, 0x40, 0xc001a65c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001a13e80, 0x7fffffff, 0xc0e5c1, 0xc002975260, 0x58, 0x58, 0xc002a96060, 0xc0016eedd0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001a13e80, 0xc0037f6400, 0x0, 0x0, 0x7fffffff, 0xc002c83b98, 0x4886820, 0x72b56a8, 0x40, 0xc0053e37c5, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001a13e80, 0x7f6d0c1990d0, 0xc000292db0, 0xc0037f6400, 0x0, 0x0, 0x3de9580, 0xc003c7f200, 0x7fffffff, 0xc002c83b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001c6b200, 0x3de9580, 0xc003c7f200, 0xc002c83b98, 0x0, 0xc003c7f200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001c6b200, 0x5, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001f66120, 0xc002c83ce0, 0xc002c83cd0, 0xc00415f2c0, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001f66120, 0x3de9580, 0xc003c7f200, 0xc00682a260, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0015a8ae0, 0xc00682a260, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc003505230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a89b0, 0xc0019e4510, 0xc001a13a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1603 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001bdca00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001bd4de0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001084000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2832 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0033ec5c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1521 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0009ed080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1520 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001bdc9c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc000f4e480, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0009ed080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2389 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc001582e10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037d450, 0xc0019e4510, 0xc00030ffc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2021 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606ae0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003115b0, 0xc0019e4510, 0xc0003115a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1604 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc0007844d0, 0x489e1a0, 0xc00116c780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b4a50, 0xc000937c20, 0xc0019b4a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1501 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1(0x489e1a0, 0xc000969f80) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:246 +0x193 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b71b0, 0xc000937c20, 0xc001dc5a60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2084 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc001460780, 0x1, 0xc002258ea0, 0xc000394b40, 0x48fa520, 0xc001d68070, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001bf7470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037dee0, 0xc000937c20, 0xc001bf7410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2140 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001234700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2023 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606b40) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311600, 0xc0019e4510, 0xc0003115e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1525 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001234000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1868 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001d8dd80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00293c060, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001892180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1528 [semacquire]: sync.runtime_SemacquireMutex(0xc0003f2464, 0x7f6d0c274600) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc0003f2460) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc0003f2460, 0xc000f46750) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc001d6d080) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68e70, 0xc000937c20, 0xc001d2fd20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1564 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001785d40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1527 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001d644c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001d752c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001234000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1526 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228d78, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00007d918, 0x72, 0x3a00, 0x3a54, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00007d900, 0xc003534000, 0x3a54, 0x3a54, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00007d900, 0xc003534000, 0x3a54, 0x3a54, 0x203001, 0x0, 0x3a47) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00030c018, 0xc003534000, 0x3a54, 0x3a54, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0071a9880, 0xc003534000, 0x3a54, 0x3a54, 0x1f20, 0x482c600, 0xc001419a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001d0a258, 0x482c340, 0xc0071a9880, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001d0a000, 0x48317e0, 0xc00030c018, 0x5, 0xc00030c018, 0x1f03) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001d0a000, 0x40e5100, 0xc001d0a138, 0xc001419d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001d0a000, 0xc000bd2000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001d751a0, 0xc00029a8f8, 0x9, 0x9, 0xc1c86c, 0xc001234168, 0xc001a1ed20) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001d751a0, 0xc00029a8f8, 0x9, 0x9, 0x9, 0x1efa, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00029a8f8, 0x9, 0x9, 0x482c140, 0xc001d751a0, 0x0, 0x1efa00000000, 0x100000003, 0x300001efa) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00029a8c0, 0xc001a1ec90, 0xc001a1ec90, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001234000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1997 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b97a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d565f0, 0xc0019e4510, 0xc00039ecc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2022 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606b10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003115d0, 0xc0019e4510, 0xc0003115c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2019 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606a20) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311570, 0xc0019e4510, 0xc000311550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1992 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b9650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56520, 0xc0019e4510, 0xc00039eb80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2174 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00174c380, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e120, 0xc00010c010, 0xc00192a1b0, 0x22, 0x0, 0x0, 0xc0010fffb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1991 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).runCloser(0xc0005c8230, 0x489e1a0, 0xc00195e390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:134 +0x333 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56110, 0xc0019e4510, 0xc000d56100) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1375 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228fe8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001580498, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001580480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001580480, 0xc0032be501, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc001d66450, 0xc001533e80, 0xc001533e88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc001d66450, 0x40e2690, 0xc000be5040, 0x48faa00, 0xc0032be5a8) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000be5040, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc0003f2460, 0xc000f42f50) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc001a1e0c0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d681c0, 0xc000937c20, 0xc001d2e1a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1498 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func1(0x489e1a0, 0xc000be6060) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:297 +0x12b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b7180, 0xc000937c20, 0xc000969a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1539 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0002dd270, 0xc001914830, 0x5, 0x5, 0x4fe4, 0xc002caa300, 0x9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0002dd270, 0xc001914830, 0x5, 0x5, 0xc11b10, 0xc0000762a0, 0xc0043b5970) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0008f31a0, 0xc001914830, 0x5, 0x5, 0x0, 0xc0043b5998, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0008f31a0, 0xc001914830, 0x5, 0x5, 0x5, 0x14dfbb6, 0xc002caa301, 0xc0043b5a50) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0011ca400, 0xc001914830, 0x5, 0x5, 0xc00309e610, 0x0, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001914820, 0x7fffffff, 0xc0071a9900, 0x3, 0x4, 0x3, 0x4, 0x9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001914820, 0xc0011ca400, 0x0, 0x0, 0x7fffffff, 0xc0043b5cb0, 0x4886820, 0x72b56a8, 0x0, 0xa, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001914820, 0x7f6d0c1990d0, 0xc000292db0, 0xc0011ca400, 0x0, 0x0, 0x3dd1ee0, 0xc003da5ae0, 0x7fffffff, 0xc0043b5cb0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0019f6000, 0x3dd1ee0, 0xc003da5ae0, 0xc0043b5cb0, 0xc0043b5d30, 0xc003da5ae0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0019f6000, 0xc0043b5d40, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001c54360, 0xc0043b5df8, 0xc0043b5de8, 0x0, 0x48) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001c54360, 0x3dd1ee0, 0xc003da5ae0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc0019b4100, 0x489e1a0, 0xc0008f3290, 0xc00193c780) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:553 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc000a9c600, 0x48f9ec0, 0xc0019b4100, 0xc001af41a0, 0x489e1a0, 0xc0008f3290, 0xc00193c780, 0x30, 0x3d2f600) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:322 +0x4a github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x489e1a0, 0xc0008f3290) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:337 +0xde github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b41a0, 0xc000937c20, 0xc001bdc3c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1540 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc000c26200, 0x48f9f20, 0xc0019b4300, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:193 +0x625 github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e0c0, 0x0, 0xc001a21200) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:569 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e0c0, 0xc001914a40, 0x40df160, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e0c0, 0xc001914a40, 0x40df160, 0x489e1a0, 0xc0008f34a0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc00166f800, 0xc0011ca500, 0xc000be7800, 0x6f9bba0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc00166f800, 0xc0011ca500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001d06910, 0xc000c12a80, 0x48fef60, 0xc00166f800, 0xc0011ca500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1584 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c88) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf76b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf76b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363310, 0xc000937c20, 0xc000363250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1541 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0002dd680, 0xc001914a30, 0x5, 0x5, 0x32, 0xc002a2d868, 0xd1dafb) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0002dd680, 0xc001914a30, 0x5, 0x5, 0xc11b10, 0xc000076320, 0xc002a2d888) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0008f3410, 0xc001914a30, 0x5, 0x5, 0x6f533a, 0xc002a2d8b0, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0008f3410, 0xc001914a30, 0x5, 0x5, 0x5, 0xc002a2d968, 0xf02137, 0x3a62440) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0011ca500, 0xc001914a30, 0x5, 0x5, 0x7ee108, 0xc0008f33b0, 0x3bbf7c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001914a20, 0x7fffffff, 0x71fd21, 0x40e43b8, 0xc002a2da08, 0x20, 0xc001751680, 0x40e43b8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001914a20, 0xc0011ca500, 0x0, 0x0, 0x7fffffff, 0xc002a2dbc0, 0x4886820, 0x72b56a8, 0x8, 0x6f8939, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001914a20, 0x7f6d0c1990d0, 0xc000292db0, 0xc0011ca500, 0x0, 0x0, 0x3dd1dc0, 0xc006bcd8b0, 0x7fffffff, 0xc002a2dbc0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001d4e0c0, 0x3dd1dc0, 0xc006bcd8b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc0019b4300, 0x768c8d, 0xc26248, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:588 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc000c26200, 0x489e1a0, 0xc0008f3680, 0xc001dcc068, 0xc001bdc540, 0xc002a2df50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:334 +0x341 github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x489e1a0, 0xc0008f3680) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:141 +0x9e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b4330, 0xc000a30090, 0xc001bdc580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2020 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606ab0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311590, 0xc0019e4510, 0xc000311580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2024 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606ba0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311640, 0xc0019e4510, 0xc000311610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1500 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc000be6510) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b71a0, 0xc000937c20, 0xc000969a70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1585 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf76e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf76e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363350, 0xc000937c20, 0xc000363320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1152 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002d8d880, 0xc000d558c0, 0x489e1a0, 0xc001799710) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1153 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0031ee5f0, 0xc0019e0ef0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0031ee5f0, 0xc0019e0ef0, 0x5, 0x5, 0xc11b10, 0xc001d73680, 0xc004abd978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001799830, 0xc0019e0ef0, 0x5, 0x5, 0x74cff0, 0xc004abd9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001799830, 0xc0019e0ef0, 0x5, 0x5, 0x5, 0xf, 0xc0011f5620, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00244c800, 0xc0019e0ef0, 0x5, 0x5, 0x83e500, 0xc007d8cbc0, 0xc004abda78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0019e0ee0, 0x7fffffff, 0x3c06120, 0xc0011f5620, 0xc004abda0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0019e0ee0, 0xc00244c800, 0x0, 0x0, 0x7fffffff, 0xc004abdcb8, 0x4886820, 0x72b56a8, 0x0, 0xc00031b5e0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0019e0ee0, 0x7f6d0c1990d0, 0xc000292db0, 0xc00244c800, 0x0, 0x0, 0x3de8800, 0xc006643b80, 0x7fffffff, 0xc004abdcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000f7b400, 0x3de8800, 0xc006643b80, 0xc004abdcb8, 0xc006d63d10, 0xc006643b80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000f7b400, 0xc000087470, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d558c0, 0xc004abde00, 0xc004abddf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d558c0, 0x3de8800, 0xc006643b80, 0x2c5e8e0, 0xc00084c700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0015a9430, 0x489e1a0, 0xc0017998c0, 0xc005b320c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0015a9430, 0xc001db8b80, 0xc0018a82d0, 0x489e1a0, 0xc0017998c0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0017998c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a9450, 0xc0019e4510, 0xc001799860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1514 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001d8c400, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001d75620, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00166f800) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1115 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1cea78, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001568f98, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001568f80, 0xc00346e800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001568f80, 0xc00346e800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e62548, 0xc00346e800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00565d7c0, 0xc00346e800, 0x791, 0x791, 0x6e, 0x482c600, 0xc002f0ba78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00113a5d8, 0x482c340, 0xc00565d7c0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00113a380, 0x48317e0, 0xc002e62548, 0x5, 0xc002e62548, 0xc00574a3f0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00113a380, 0x40e5100, 0xc00113a4b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00113a380, 0xc00317a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003582300, 0xc003442038, 0x9, 0x9, 0xc0028f0a78, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003582300, 0xc003442038, 0x9, 0x9, 0x9, 0xc005bbba70, 0x485e320, 0xc007011f00) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc003442038, 0x9, 0x9, 0x482c140, 0xc003582300, 0x0, 0xc000000000, 0xc001633d00, 0x574a300) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc003442000, 0xc00574a3c0, 0xc00574a3c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc003192000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1982 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0021c2000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2797 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce9a8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001569018, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001569000, 0xc00346f000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001569000, 0xc00346f000, 0x75b, 0x75b, 0xc001833938, 0x701, 0xc00113af80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e62558, 0xc00346f000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00311b920, 0xc00346f000, 0x75b, 0x75b, 0xc00346f00d, 0x8e, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0048d3960, 0xc00346f000, 0x75b, 0x75b, 0xab, 0x482c600, 0xc001833a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc00113b058, 0x482c340, 0xc0048d3960, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc00113ae00, 0x7f6d0700c008, 0xc00311b920, 0x5, 0xc00311b920, 0x8e) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc00113ae00, 0x40e5100, 0xc00113af38, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc00113ae00, 0xc00343a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002638240, 0xc001604658, 0x9, 0x9, 0xc001d8df48, 0xc004e2e630, 0x85) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002638240, 0xc001604658, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc001604658, 0x9, 0x9, 0x482c140, 0xc002638240, 0x0, 0x300000000, 0xc0037f6200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc001604620, 0xc00333c330, 0xc00333c330, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0022d5200, 0xc00333c5a0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc0022d5200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc0022d5200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1909 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc001460780, 0x48fa580, 0xc000278430, 0x485c120, 0xc001460780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc001810000, 0x0, 0xc001894840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc001810000, 0xc001a12300, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc001810000, 0xc001a12300, 0x40e1ba8, 0x489e1a0, 0xc00185cd20) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc001892180, 0xc002d1e000, 0xc00111b2c0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc001892180, 0xc002d1e000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000318e10, 0xc00193c600, 0x48fef60, 0xc001892180, 0xc002d1e000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1117 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc000383de0, 0x7f6d05d41348, 0xc0017f4110, 0x7f6d05d41348, 0xc0017f4110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc000383de0, 0x48fa760, 0xc0017f4110, 0xc02f00010301, 0xc0017f4110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc00241c000, 0x0, 0xc002e18a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc00241c000, 0xc00333e300, 0x40e1e60, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc00241c000, 0xc00333e300, 0x40e1e60, 0x489e1a0, 0xc0031c0480) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0011ca100, 0xc000c30750, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0011ca100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002c1090, 0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0011ca100) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 1981 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0009cbd40, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0016a2ba0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0021c2000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2380 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001785b00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2162 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002d8a180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2082 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3308, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a918, 0x72, 0x3a00, 0x3a54, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000f7a900, 0xc003282000, 0x3a54, 0x3a54, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000f7a900, 0xc003282000, 0x3a54, 0x3a54, 0x203001, 0x0, 0x3a47) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f94878, 0xc003282000, 0x3a54, 0x3a54, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0077a2b60, 0xc003282000, 0x3a54, 0x3a54, 0x1f20, 0x482c600, 0xc002519a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001ce2cd8, 0x482c340, 0xc0077a2b60, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001ce2a80, 0x48317e0, 0xc000f94878, 0x5, 0xc000f94878, 0x1f03) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001ce2a80, 0x40e5100, 0xc001ce2bb8, 0xc002519d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001ce2a80, 0xc0029ce000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002258a20, 0xc000fcec78, 0x9, 0x9, 0xc1c86c, 0xc00293e4e8, 0xc001bf61e0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002258a20, 0xc000fcec78, 0x9, 0x9, 0x9, 0x1efa, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000fcec78, 0x9, 0x9, 0x482c140, 0xc002258a20, 0x0, 0x1efa00000000, 0x100000003, 0x300001efa) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000fcec40, 0xc001bf60f0, 0xc001bf60f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00293e380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2499 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0002a8960, 0x48fa580, 0xc0002fa150, 0x485c120, 0xc0002a8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc003366180, 0x0, 0xc002e18a80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc003366180, 0xc00031a620, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc003366180, 0xc00031a620, 0x40e1ba8, 0x489e1a0, 0xc0017f0570) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0015c4500, 0xc000c58270, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0015c4500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002c1090, 0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc0015c4500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2170 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2ef8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000ba2998, 0x72, 0x6900, 0x6967, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000ba2980, 0xc004243500, 0x6967, 0x6967, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000ba2980, 0xc004243500, 0x6967, 0x6967, 0xc0013fe938, 0x6901, 0xc002d5e500) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0012503f0, 0xc004243500, 0x6967, 0x6967, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002d50c00, 0xc004243500, 0x6967, 0x6967, 0xc00424350d, 0x5a, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0064cfa20, 0xc004243500, 0x6967, 0x6967, 0x77, 0x482c600, 0xc0013fea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002d5e5d8, 0x482c340, 0xc0064cfa20, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002d5e380, 0x7f6d0700c008, 0xc002d50c00, 0x5, 0xc002d50c00, 0x5a) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002d5e380, 0x40e5100, 0xc002d5e4b8, 0x3b) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002d5e380, 0xc002e76000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002e18ae0, 0xc000226e38, 0x9, 0x9, 0xc0031984c8, 0xc0030e9ec0, 0x51) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002e18ae0, 0xc000226e38, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000226e38, 0x9, 0x9, 0x482c140, 0xc002e18ae0, 0x0, 0x3b00000000, 0xc0015c4500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000226e00, 0xc0016d45d0, 0xc0016d45d0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001c10900, 0xc0016d46f0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc001c10900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc001c10900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2163 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3098, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001023118, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001023100, 0xc002d96000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001023100, 0xc002d96000, 0x75b, 0x75b, 0xc0019a3938, 0x701, 0xc0022d2880) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00030c4c8, 0xc002d96000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0014c39e0, 0xc002d96000, 0x75b, 0x75b, 0xc002d9600d, 0x64, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc004caa4a0, 0xc002d96000, 0x75b, 0x75b, 0x81, 0x482c600, 0xc0019a3a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0022d2958, 0x482c340, 0xc004caa4a0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0022d2700, 0x7f6d0700c008, 0xc0014c39e0, 0x5, 0xc0014c39e0, 0x64) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0022d2700, 0x40e5100, 0xc0022d2838, 0xb) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0022d2700, 0xc002dcc000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002c855c0, 0xc00111d378, 0x9, 0x9, 0xc001a2ea88, 0xc003399ec0, 0x5b) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002c855c0, 0xc00111d378, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111d378, 0x9, 0x9, 0x482c140, 0xc002c855c0, 0x0, 0xb00000000, 0xc001633600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111d340, 0xc001d8bb30, 0xc001d8bb30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002d8a180, 0xc000930270, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc002d8a180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc002d8a180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1985 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018cacc0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002c85620, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc002d8a180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2100 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001d778b0, 0xc00036f870, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001d778b0, 0xc00036f870, 0x5, 0x5, 0xc11b10, 0xc0005e0880, 0xc0038b3978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001d547e0, 0xc00036f870, 0x5, 0x5, 0x74cff0, 0xc0038b39a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001d547e0, 0xc00036f870, 0x5, 0x5, 0x5, 0xf, 0xc000079c80, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001a39200, 0xc00036f870, 0x5, 0x5, 0x83e500, 0xc007011b80, 0xc0038b3a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00036f860, 0x7fffffff, 0x3c06120, 0xc000079c80, 0xc0038b3a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00036f860, 0xc001a39200, 0x0, 0x0, 0x7fffffff, 0xc0038b3cb8, 0x4886820, 0x72b56a8, 0x0, 0xc004d9f300, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00036f860, 0x7f6d0c1990d0, 0xc000292db0, 0xc001a39200, 0x0, 0x0, 0x3de8800, 0xc005514e00, 0x7fffffff, 0xc0038b3cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000f7ad00, 0x3de8800, 0xc005514e00, 0xc0038b3cb8, 0xc0027d0410, 0xc005514e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000f7ad00, 0xc000084f70, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001b1aea0, 0xc0038b3e00, 0xc0038b3df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001b1aea0, 0x3de8800, 0xc005514e00, 0x2c5e8e0, 0xc000a15500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0003426c0, 0x489e1a0, 0xc001d54870, 0xc002fbf200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0003426c0, 0xc000394e80, 0xc0002a8960, 0x489e1a0, 0xc001d54870, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc001d54870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003426e0, 0xc000a30090, 0xc001d54810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 934 [chan receive]: github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).execInternal(0xc000c36d80, 0x489e1a0, 0xc006eb32c0, 0x3f85606, 0x9, 0xc005fc9950, 0x1, 0x3f7ae7b, 0x4, 0xc006eb32f0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:565 +0x9ce github.com/cockroachdb/cockroach/pkg/sql.(*InternalExecutor).Exec(0xc000c36d80, 0x489e0e0, 0xc0064f58c0, 0x3f85606, 0x9, 0xc005fc9950, 0x4047590, 0x7d, 0xc007c9b7b8, 0x4, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:300 +0xfa github.com/cockroachdb/cockroach/pkg/sql.EventLogger.InsertEventRecord(0xc000c36d80, 0x489e0e0, 0xc0064f58c0, 0xc005fc9950, 0x3f92879, 0xf, 0x100000060, 0x3c0ebe0, 0xc006eb3260, 0x3e95f20, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/event_log.go:160 +0x221 github.com/cockroachdb/cockroach/pkg/sql.(*createDatabaseNode).startExec(0xc00075c490, 0x489e0e0, 0xc0064f58c0, 0xc0043d1d40, 0xc000f5c358, 0x6, 0xc007c9b970) /go/src/github.com/cockroachdb/cockroach/pkg/sql/create_database.go:85 +0x303 github.com/cockroachdb/cockroach/pkg/sql.startExec.func2(0x3f9281f, 0xf, 0x489f1e0, 0xc00075c490, 0xc004184000, 0xc007c9b980) /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:362 +0x55 github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal.func1(0xc00636cba0, 0x3f9281f, 0xf, 0x489f1e0, 0xc00075c490) /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:141 +0x5d github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visitInternal(0xc00636cba0, 0x489f1e0, 0xc00075c490, 0x3f9281f, 0xf) /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:668 +0x286 github.com/cockroachdb/cockroach/pkg/sql.(*planVisitor).visit(0xc00636cba0, 0x489f1e0, 0xc00075c490, 0x0, 0x60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:108 +0x8d github.com/cockroachdb/cockroach/pkg/sql.walkPlan(...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/walk.go:72 github.com/cockroachdb/cockroach/pkg/sql.startExec(0x489e0e0, 0xc0064f58c0, 0xc0043d1d40, 0xc000f5c358, 0x489f1e0, 0xc00075c490, 0x0, 0x72b7b40) /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan.go:365 +0x286 github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start(0xc0006a2500, 0x489e0e0, 0xc0064f58c0, 0x72b7b40, 0x39ac120) /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_node_to_row_source.go:117 +0xd6 github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*ProcessorBase).Run(0xc0006a2500, 0x489e0e0, 0xc0064f58c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/processors.go:792 +0x52 github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*Flow).Run(0xc002b24900, 0x489e1a0, 0xc0020e7020, 0x40df8c8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow.go:584 +0x1ec github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run(0xc0004aef00, 0xc00636ca80, 0xc005fc9950, 0xc007c9c760, 0xc007228840, 0xc000f5c428, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:337 +0x381 github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun(0xc0004aef00, 0x489e1a0, 0xc0020e6f00, 0xc000f5c428, 0xc00636ca80, 0xc005fc9950, 0x489f1e0, 0xc00075c490, 0xc007228840, 0xc000f5c1d8) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:962 +0x20a github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine(0xc000f5c000, 0x489e1a0, 0xc0020e6f00, 0xc000f5c358, 0x1, 0x7f6d05dc26e8, 0xc0015ea668, 0xc007c9c900, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:816 +0x370 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine(0xc000f5c000, 0x489e1a0, 0xc0020e6f00, 0xc000f5c358, 0x7f6d05dc26e8, 0xc0015ea668, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:708 +0x6de github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState(0xc000f5c000, 0x489e1a0, 0xc0020e6f00, 0x48c1660, 0xc003b19080, 0xc0028f2c78, 0x14, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:417 +0xc2f github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt(0xc000f5c000, 0x489e1a0, 0xc0020e6f00, 0x48c1660, 0xc003b19080, 0xc0028f2c78, 0x14, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:98 +0x4ec github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc000f5c000, 0x489e0e0, 0xc0064f5800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1238 +0x1aa2 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc000f5c000, 0x489e0e0, 0xc00146c2c0, 0xc000788c40, 0x5400, 0x15000, 0xc000788cd8, 0xc0002604b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc00042b400, 0x489e0e0, 0xc00146c2c0, 0xc000f5c000, 0x5400, 0x15000, 0xc000788cd8, 0xc0002604b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0002f08f1, 0xc00031ab20, 0x489e0e0, 0xc00146c2c0, 0xc0002604b0, 0xc00042b400, 0xc0015ea000, 0x48c0ea0, 0xc00031ab00, 0xc000a95260, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 933 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c229188, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000ba2098, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000ba2080, 0xc001c1b800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000ba2080, 0xc001c1b800, 0x719, 0x719, 0x203000, 0x10000, 0xc0b9311e97) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001dcc020, 0xc001c1b800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc0036341c0, 0xc001c1b800, 0x719, 0x719, 0xd1dd5e, 0xc002eb0780, 0xc003ad9618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000c7e258, 0x482c340, 0xc0036341c0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000c7e000, 0x48317e0, 0xc001dcc020, 0x5, 0xc001dcc020, 0xc003fda800) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000c7e000, 0x40e5100, 0xc000c7e138, 0xed500c079) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000c7e000, 0xc000f56000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc00031aac0, 0xc000f56000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc0015ea048) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc0015ea048, 0xc0015ea620, 0xed500c078, 0x0) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc0015ea7a0, 0x486e520, 0xc0015ea048, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc0015ea000, 0x489e0e0, 0xc00146c000, 0xc000260490, 0xc00042b400, 0x5400, 0x15000, 0xc000788cd8, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc00146c000, 0x48f9e60, 0xc000c7e000, 0xc000b0f01b, 0x4, 0xc0009a8b40, 0x485eca0, 0xc0009a84b0, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc000788b40, 0x489e0e0, 0xc00146c000, 0x48f9e60, 0xc000c7e000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc001dcc020) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc000a30090, 0x489e1a0, 0xc0009a8420, 0xc0016687a8, 0x48faa00, 0xc001dcc020, 0xc0009a8450) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 2085 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001dff880, 0xc0015bd680, 0x489e1a0, 0xc001bf7560) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 236806 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6e, 0x490bae0, 0xc002c84000, 0x0, 0x1, 0x38, 0x50) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6e, 0xc0063447e0, 0xc005af8000, 0x15c106cd3e1609ac) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0069e1e00, 0x33, 0x0, 0x0, 0xc005022c00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1910 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc001878140, 0xc001a122f0, 0x5, 0x5, 0x0, 0x0, 0xc0056b0200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001878140, 0xc001a122f0, 0x5, 0x5, 0xc11b10, 0xc001d72020, 0xc002d3daf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00185c870, 0xc001a122f0, 0x5, 0x5, 0x39, 0xc002d3db20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00185c870, 0xc001a122f0, 0x5, 0x5, 0x5, 0xc002d3dbd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002d1e000, 0xc001a122f0, 0x5, 0x5, 0x203000, 0x203000, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001a122e0, 0x7fffffff, 0x1fa90c6d19c7fc03, 0x1fa90c6d19c7fc03, 0xc002d3dc78, 0x2c15de0, 0x3a5c620, 0xc0005ac090) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001a122e0, 0xc002d1e000, 0x0, 0x0, 0x7fffffff, 0xc002d3de30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001a122e0, 0x7f6d0c1990d0, 0xc000292db0, 0xc002d1e000, 0x0, 0x0, 0x3de86e0, 0xc001dc4680, 0x7fffffff, 0xc002d3de30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001810000, 0x3de86e0, 0xc001dc4680, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc000278430, 0x489e1a0, 0xc00185ce10, 0xc0056b0200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc000278430, 0xc001460780, 0x489e1a0, 0xc00185ce10, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc00185ce10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000278450, 0xc000937c20, 0xc00185cde0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1889 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018aeac0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001d00e40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0010841c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2168 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0017a1640, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002e18b40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001c10900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1885 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc001d88ff0, 0xc001d6b070, 0x5, 0x5, 0x32, 0xc002efb868, 0xd1dafb) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001d88ff0, 0xc001d6b070, 0x5, 0x5, 0xc11b10, 0xc001d116a0, 0xc002efb888) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001607d40, 0xc001d6b070, 0x5, 0x5, 0x6f533a, 0xc002efb8b0, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001607d40, 0xc001d6b070, 0x5, 0x5, 0x5, 0xc002efb968, 0xf02137, 0x3a62440) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000b24500, 0xc001d6b070, 0x5, 0x5, 0x7ee108, 0xc001607ce0, 0x3bbf7c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc001d6b060, 0x7fffffff, 0x71fd21, 0x40e43b8, 0xc002efba08, 0x20, 0xc001954600, 0x40e43b8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc001d6b060, 0xc000b24500, 0x0, 0x0, 0x7fffffff, 0xc002efbbc0, 0x4886820, 0x72b56a8, 0x8, 0x6f8939, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc001d6b060, 0x7f6d0c1990d0, 0xc000292db0, 0xc000b24500, 0x0, 0x0, 0x3dd1dc0, 0xc006bcd810, 0x7fffffff, 0xc002efbbc0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001d4e000, 0x3dd1dc0, 0xc006bcd810, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipServer).Recv(0xc000d566e0, 0x768c8d, 0xc26248, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:588 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*server).gossipReceiver(0xc000c26200, 0x489e1a0, 0xc001d54270, 0xc001668058, 0xc0018ae180, 0xc002efbf50, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:334 +0x341 github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip.func3.1(0x489e1a0, 0xc001d54270) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:141 +0x9e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56710, 0xc000a30090, 0xc0018ae200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2184 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002c8c2c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2083 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0015f3c40, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0016a3200, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00293e380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2209 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc001582ed0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037d380, 0xc0019e4510, 0xc00030ff80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1993 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b9680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56540, 0xc0019e4510, 0xc00039eba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1888 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3168, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001023098, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001023080, 0xc001c1a800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001023080, 0xc001c1a800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00075c090, 0xc001c1a800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc007c864a0, 0xc001c1a800, 0x791, 0x791, 0x73, 0x482c600, 0xc001416a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0022d25d8, 0x482c340, 0xc007c864a0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0022d2380, 0x48317e0, 0xc00075c090, 0x5, 0xc00075c090, 0x56) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0022d2380, 0x40e5100, 0xc0022d24b8, 0xc001416d40) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0022d2380, 0xc002de4000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001d00d80, 0xc000375df8, 0x9, 0x9, 0xc1c86c, 0xc001084328, 0xc001d6d8f0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001d00d80, 0xc000375df8, 0x9, 0x9, 0x9, 0x4d, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000375df8, 0x9, 0x9, 0x482c140, 0xc001d00d80, 0x0, 0x4d00000000, 0x10000000b, 0xb0000004d) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000375dc0, 0xc001d6d860, 0xc001d6d860, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0010841c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1979 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc001607200) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001dbd870, 0xc0019e4510, 0xc00038b4c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1983 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3238, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7aa18, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000f7aa00, 0xc00285e800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000f7aa00, 0xc00285e800, 0x75b, 0x75b, 0xc001411938, 0x701, 0xc001ce2f80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f94888, 0xc00285e800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0018955c0, 0xc00285e800, 0x75b, 0x75b, 0xc00285e80d, 0x19d, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0048d26e0, 0xc00285e800, 0x75b, 0x75b, 0x1ba, 0x482c600, 0xc001411a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001ce3058, 0x482c340, 0xc0048d26e0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001ce2e00, 0x7f6d0700c008, 0xc0018955c0, 0x5, 0xc0018955c0, 0x19d) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001ce2e00, 0x40e5100, 0xc001ce2f38, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001ce2e00, 0xc0021ba000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001d00000, 0xc000374ff8, 0x9, 0x9, 0xc0017a0288, 0xc0083ee820, 0x194) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001d00000, 0xc000374ff8, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000374ff8, 0x9, 0x9, 0x482c140, 0xc001d00000, 0x0, 0x300000000, 0xc000b24500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000374fc0, 0xc00079f8f0, 0xc00079f8f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0021c2000, 0xc0016079b0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc0021c2000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc0021c2000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1495 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001ca4300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1494 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000be4680, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001c7e720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc001ca4300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1516 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228e48, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001d92018, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001d92000, 0xc001e99800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001d92000, 0xc001e99800, 0x75b, 0x75b, 0xc001e87938, 0x701, 0xc001dfe180) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00075c018, 0xc001e99800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0019b6000, 0xc001e99800, 0x75b, 0x75b, 0xc001e9980d, 0x19e, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc00449cb00, 0xc001e99800, 0x75b, 0x75b, 0x1bb, 0x482c600, 0xc001e87a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001dfe258, 0x482c340, 0xc00449cb00, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001dfe000, 0x7f6d0700c008, 0xc0019b6000, 0x5, 0xc0019b6000, 0x19e) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001dfe000, 0x40e5100, 0xc001dfe138, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001dfe000, 0xc000d58000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001a21260, 0xc0003e0658, 0x9, 0x9, 0xc001bdc488, 0xc004e464e0, 0x195) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001a21260, 0xc0003e0658, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003e0658, 0x9, 0x9, 0x482c140, 0xc001a21260, 0x0, 0x300000000, 0xc0011ca500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e0620, 0xc001c94c00, 0xc001c94c00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00166f800, 0xc001c94d20, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc00166f800) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc00166f800) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1496 [select]: net.(*pipe).read(0xc001580600, 0xc00034f800, 0x75b, 0x75b, 0x203001, 0x0, 0x74e) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc001580600, 0xc00034f800, 0x75b, 0x75b, 0xc00034f80d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc004742e60, 0xc00034f800, 0x75b, 0x75b, 0x2e, 0x482c600, 0xc001464a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001e7b058, 0x482c340, 0xc004742e60, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001e7ae00, 0x7f6d0c22ebd8, 0xc001580600, 0x5, 0xc001580600, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001e7ae00, 0x40e5100, 0xc001e7af38, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001e7ae00, 0xc000872000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001c7e600, 0xc0016a03b8, 0x9, 0x9, 0xc001a65c00, 0x7f6d0c2746d0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001c7e600, 0xc0016a03b8, 0x9, 0x9, 0x9, 0xbdd055, 0xc004f3774c, 0xc001a1bc48) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0016a03b8, 0x9, 0x9, 0x482c140, 0xc001c7e600, 0x0, 0x0, 0xc004f37740, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0016a0380, 0xc004f37740, 0xc004f37740, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001ca4300, 0xc001606f90, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc001ca4300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc001ca4300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2099 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0015a6e00, 0xc001b1aea0, 0x489e1a0, 0xc001d546c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2150 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc002d1db60, 0xc000448ed0, 0xf, 0xc000a9d1a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0016ba6f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000279130, 0xc0019e4510, 0xc002e1e140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1828 [select]: database/sql.(*DB).connectionOpener(0xc001c44180, 0x489e0e0, 0xc000f906c0) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 2886 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000c22000, 0xc002e65e60, 0xc001ae6320, 0xf, 0xc002975620, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001b2b1d0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5160, 0xc000a30090, 0xc0034a5040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1832 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228898, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001c6ad18, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001c6ad00, 0xc00285e000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001c6ad00, 0xc00285e000, 0x719, 0x719, 0x203001, 0x48314e0, 0x72b4ea0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001d66198, 0xc00285e000, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00412d7c0, 0xc00285e000, 0x719, 0x719, 0x0, 0x48317a0, 0xc002f15618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0018f3058, 0x482c340, 0xc00412d7c0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0018f2e00, 0x48317e0, 0xc001d66198, 0x5, 0xc001d66198, 0xc0056a2c00) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0018f2e00, 0x40e5100, 0xc0018f2f38, 0xed500c078) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0018f2e00, 0xc0028f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc00180e340, 0xc0028f3000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc0015ea948) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc0015ea948, 0x1, 0x1, 0x2) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc0015eb0a0, 0x486e520, 0xc0015ea948, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc0015ea900, 0x489e0e0, 0xc000f913c0, 0xc001898bc0, 0xc0006a2a00, 0x5400, 0x15000, 0xc001a5d158, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc000f913c0, 0x48f9e60, 0xc0018f2e00, 0xc001e1b058, 0x4, 0xc0015fdd40, 0x485eca0, 0xc000c3d3b0, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc001a5cfc0, 0x489e0e0, 0xc000f913c0, 0x48f9e60, 0xc0018f2e00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc001d66198) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc000937c20, 0x489e1a0, 0xc000c3d320, 0xc001d66100, 0x48faa00, 0xc001d66198, 0xc000c3d350) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 2388 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func3(0x489e1a0, 0xc001582f60) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:322 +0xfb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00037d440, 0xc0019e4510, 0xc001582a80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 5674 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002daedc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2571 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc00039e140, 0x7f6d05d41348, 0xc001c0f080, 0x7f6d05d41348, 0xc001c0f080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc00039e140, 0x48fa760, 0xc001c0f080, 0x108, 0x109) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00048ba40, 0x0, 0xc002e646c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00048ba40, 0xc001b35ec0, 0x40e1e60, 0x3c89ac0, 0x47cf94f) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00048ba40, 0xc001b35ec0, 0x40e1e60, 0x489e1a0, 0xc001a9b530) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc003110000, 0xc002d1fa00, 0xc000c3c270, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc003110000, 0xc002d1fa00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003abbd0, 0xc00262b200, 0x48fef60, 0xc003110000, 0xc002d1fa00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2151 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002dae840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 236607 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x30, 0x490bae0, 0xc002e37e60, 0x0, 0x1, 0x0, 0x706f17) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x30, 0xc0061e6900, 0xc000944800, 0x15c106cd3b4863c6) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc004ec8e80, 0x33, 0x0, 0x0, 0xc001a376b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 3103 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc001460780, 0x1, 0xc00325b560, 0xc002a4c240, 0x48fa520, 0xc001842fb0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0032c1ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001842f40, 0xc000937c20, 0xc0032c1aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3041 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce8d8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001c6b418, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001c6b400, 0xc0025b8000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001c6b400, 0xc0025b8000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001668878, 0xc0025b8000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc007125440, 0xc0025b8000, 0x791, 0x791, 0x42, 0x482c600, 0xc0024f8a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001c5acd8, 0x482c340, 0xc007125440, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001c5aa80, 0x48317e0, 0xc001668878, 0x5, 0xc001668878, 0xc007693800) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001c5aa80, 0x40e5100, 0xc001c5abb8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001c5aa80, 0xc003042000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002612840, 0xc002ffc1f8, 0x9, 0x9, 0xc008448178, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002612840, 0xc002ffc1f8, 0x9, 0x9, 0x9, 0xc008304ff0, 0x485e320, 0xc0057285b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc002ffc1f8, 0x9, 0x9, 0x482c140, 0xc002612840, 0x0, 0xc000000000, 0xc006449300, 0x7693700) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc002ffc1c0, 0xc0076937d0, 0xc0076937d0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc003640540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 3251 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0036b1c20, 0xc00279ce10, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0036b1c20, 0xc00279ce10, 0x5, 0x5, 0xc11b10, 0xc001d72c20, 0xc001b79af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0032c1ec0, 0xc00279ce10, 0x5, 0x5, 0x0, 0xc001b79b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0032c1ec0, 0xc00279ce10, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002700f00, 0xc00279ce10, 0x5, 0x5, 0x0, 0x100000001, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00279ce00, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00279ce00, 0xc002700f00, 0x0, 0x0, 0x7fffffff, 0xc001b79e30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00279ce00, 0x7f6d0c1990d0, 0xc000292db0, 0xc002700f00, 0x0, 0x0, 0x3de86e0, 0xc004a0ea20, 0x7fffffff, 0xc001b79e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc002724480, 0x3de86e0, 0xc004a0ea20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001843080, 0x489e1a0, 0xc0032c1f80, 0xc00354ea00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001843080, 0xc0002a8960, 0x489e1a0, 0xc0032c1f80, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0032c1f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018430a0, 0xc000a30090, 0xc0032c1f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3042 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0017435c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002612900, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc003640540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2086 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001621ea0, 0xc000412cb0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001621ea0, 0xc000412cb0, 0x5, 0x5, 0xc11b10, 0xc00148c6c0, 0xc00497d978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0008f2090, 0xc000412cb0, 0x5, 0x5, 0x74cff0, 0xc00497d9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0008f2090, 0xc000412cb0, 0x5, 0x5, 0x5, 0xf, 0xc00107e1e0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00244c900, 0xc000412cb0, 0x5, 0x5, 0x83e500, 0xc0065729c0, 0xc00497da78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000412ca0, 0x7fffffff, 0x3c06120, 0xc00107e1e0, 0xc00497da0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000412ca0, 0xc00244c900, 0x0, 0x0, 0x7fffffff, 0xc00497dcb8, 0x4886820, 0x72b56a8, 0x0, 0xc006fe0340, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000412ca0, 0x7f6d0c1990d0, 0xc000292db0, 0xc00244c900, 0x0, 0x0, 0x3de8800, 0xc0030ee3c0, 0x7fffffff, 0xc00497dcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001080d00, 0x3de8800, 0xc0030ee3c0, 0xc00497dcb8, 0xc006a54640, 0xc0030ee3c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001080d00, 0xc000087470, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0015bd680, 0xc00497de00, 0xc00497ddf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0015bd680, 0x3de8800, 0xc0030ee3c0, 0x2c5e8e0, 0xc000932000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc001d68070, 0x489e1a0, 0xc0008f2150, 0xc0018cb0c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc001d68070, 0xc000394b40, 0xc001460780, 0x489e1a0, 0xc0008f2150, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0008f2150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68090, 0xc000937c20, 0xc0008f20c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2088 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc000bb60a0, 0xc000413070, 0x5, 0x5, 0x0, 0x0, 0xc00314e000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000bb60a0, 0xc000413070, 0x5, 0x5, 0xc11b10, 0xc00148c700, 0xc0053ffaf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0008f23f0, 0xc000413070, 0x5, 0x5, 0x3a, 0xc0053ffb20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0008f23f0, 0xc000413070, 0x5, 0x5, 0x5, 0xc0053ffbd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00244ca00, 0xc000413070, 0x5, 0x5, 0x203000, 0x203000, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc000413060, 0x7fffffff, 0x732a2f, 0xc000a19080, 0x4, 0xc0053ffc70, 0x733ecb, 0xc003548c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc000413060, 0xc00244ca00, 0x0, 0x0, 0x7fffffff, 0xc0053ffe30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc000413060, 0x7f6d0c1990d0, 0xc000292db0, 0xc00244ca00, 0x0, 0x0, 0x3de86e0, 0xc00380a040, 0x7fffffff, 0xc0053ffe30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc001c44240, 0x3de86e0, 0xc00380a040, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc001d68140, 0x489e1a0, 0xc0008f2570, 0xc00314e000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc001d68140, 0xc0002a8960, 0x489e1a0, 0xc0008f2570, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0008f2570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d68180, 0xc000a30090, 0xc0008f2510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2382 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0009cb890, 0x1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0009cb880) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc00157d820, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:437 +0x89 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc002c45000, 0x489e0e0, 0xc0009cb8c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1191 +0x52 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc002c45000, 0x489e0e0, 0xc0009cb8c0, 0xc001aab540, 0x5400, 0x15000, 0xc001aab5d8, 0xc0016660c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc001960f00, 0x489e0e0, 0xc0009cb8c0, 0xc002c45000, 0x5400, 0x15000, 0xc001aab5d8, 0xc0016660c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc0002c04b1, 0xc0003c3a40, 0x489e0e0, 0xc0009cb8c0, 0xc0016660c0, 0xc001960f00, 0xc00157d200, 0x48c0ea0, 0xc0003c3a20, 0xc002db49c0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 2141 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2fc8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc00160c598, 0x72, 0x4300, 0x43c3, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc00160c580, 0xc00322e000, 0x43c3, 0x43c3, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc00160c580, 0xc00322e000, 0x43c3, 0x43c3, 0x203000, 0x0, 0x43b6) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001dcd890, 0xc00322e000, 0x43c3, 0x43c3, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00039e960, 0xc00322e000, 0x43c3, 0x43c3, 0x92, 0x482c600, 0xc00213ba78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002d8de58, 0x482c340, 0xc00039e960, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002d8dc00, 0x48317e0, 0xc001dcd890, 0x5, 0xc001dcd890, 0xc006110ae0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002d8dc00, 0x40e5100, 0xc002d8dd38, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002d8dc00, 0xc002e96000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002e2c780, 0xc00111d458, 0x9, 0x9, 0xc0030e9bb8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002e2c780, 0xc00111d458, 0x9, 0x9, 0x9, 0xc0064a22a0, 0x485e320, 0xc004ff0d30) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111d458, 0x9, 0x9, 0x482c140, 0xc002e2c780, 0x0, 0xc000000000, 0xc0072b9100, 0x6110a00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111d420, 0xc006110ab0, 0xc006110ab0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001234700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1887 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0010841c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2149 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0019c07c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2136 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002c8c000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2172 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc00195e3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffb90, 0xc0019e4510, 0xc0002c83c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1459 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc00185d590) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69700, 0xc000937c20, 0xc00185ce40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1336 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c030) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8300, 0xc000937c20, 0xc0015a82f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1602 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228b08, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001d92998, 0x72, 0x2f00, 0x2fdf, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001d92980, 0xc00263a000, 0x2fdf, 0x2fdf, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001d92980, 0xc00263a000, 0x2fdf, 0x2fdf, 0x203001, 0x0, 0x2fd2) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00075c1d0, 0xc00263a000, 0x2fdf, 0x2fdf, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc006fe0dc0, 0xc00263a000, 0x2fdf, 0x2fdf, 0xfb, 0x482c600, 0xc001532a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001c805d8, 0x482c340, 0xc006fe0dc0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001c80380, 0x48317e0, 0xc00075c1d0, 0x5, 0xc00075c1d0, 0xc0075e13e0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001c80380, 0x40e5100, 0xc001c804b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001c80380, 0xc00156c000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001bd4cc0, 0xc000375a78, 0x9, 0x9, 0xc005c95858, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001bd4cc0, 0xc000375a78, 0x9, 0x9, 0x9, 0xc007143500, 0x485e320, 0xc006573120) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc000375a78, 0x9, 0x9, 0x482c140, 0xc001bd4cc0, 0x0, 0xc000000000, 0xc0075a6b00, 0x75e1300) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc000375a40, 0xc0075e13b0, 0xc0075e13b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001084000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1335 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c0c0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a82e0, 0xc000937c20, 0xc0015a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1334 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c090) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a82c0, 0xc000937c20, 0xc0015a82b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1333 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c060) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a82a0, 0xc000937c20, 0xc0015a8290) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1332 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00193ca20, 0xc000000000) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00193ca10) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc00193c900) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 1331 [select]: github.com/cockroachdb/cockroach/pkg/gossip.newInfoStore.func1(0x489e1a0, 0xc00117a330) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/infostore.go:190 +0x15b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d573d0, 0xc000937c20, 0xc000d573c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1583 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c84) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363200, 0xc000937c20, 0xc0003631b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1330 [chan receive]: github.com/cockroachdb/cockroach/pkg/rpc.NewContextWithTestingKnobs.func1(0x489e1a0, 0xc00117a300) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:478 +0x52 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d572f0, 0xc000937c20, 0xc0003fc8e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1337 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c2a0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8320, 0xc000937c20, 0xc0015a8310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2417 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000b51180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1499 [select]: github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).Start.func2(0x489e1a0, 0xc000be6090) /go/src/github.com/cockroachdb/cockroach/pkg/jobs/registry.go:308 +0xda github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b7190, 0xc000937c20, 0xc001dc5a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1538 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001b18000, 0xc001c54360, 0x489e1a0, 0xc0008f3050) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1370 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001c84720, 0xc000001bd1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001c84710) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc001c84600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 1371 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc001a1e0f0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d680b0, 0xc000937c20, 0xc001a1e030) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1339 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c2d0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8360, 0xc000937c20, 0xc0015a8350) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1340 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00156bd10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8380, 0xc000937c20, 0xc0015a8370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1341 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c120) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a83a0, 0xc000937c20, 0xc0015a8390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1342 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c150) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a83c0, 0xc000937c20, 0xc0015a83b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1343 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c180) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a83e0, 0xc000937c20, 0xc0015a83d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1344 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c1b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8400, 0xc000937c20, 0xc0015a83f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1345 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c0f0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8420, 0xc000937c20, 0xc0015a8410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1410 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c210) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8440, 0xc000937c20, 0xc0015a8430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1411 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c1e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8460, 0xc000937c20, 0xc0015a8450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1412 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00158c240) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8480, 0xc000937c20, 0xc0015a8470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1413 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0015bde60, 0x489e0e0, 0xc0015abf80) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 1414 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc0018aa000, 0x489e0e0, 0xc0015abf80) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 1460 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc00185d620) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69720, 0xc000937c20, 0xc001d69710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1461 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc000f87b80, 0xc001e1ec40, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 1462 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc00185d650) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d697d0, 0xc000937c20, 0xc001dc5f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1463 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc00185d680) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69810, 0xc000937c20, 0xc001dc5f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1464 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c2290b8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001580418, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc001580400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc001580400, 0xc001cd2000, 0x7f6d0c277648, 0xc001ca5080) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc001d66438, 0x71f47f, 0xc000526c70, 0xc00167d6f0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc001d66438, 0x40e2688, 0xc000be5000, 0xc0003f24e4, 0x4906de0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc000be5000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc00185d830) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69820, 0xc000937c20, 0xc001dc5fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1465 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc001a3fad0, 0x4886ea0, 0xc001dc5f80, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc00185d860) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69830, 0xc000937c20, 0xc00185cfc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1466 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc001dc5fe0, 0x3d2f600, 0xc00185daa0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc001a3f790, 0x486e6e0, 0xc001dc5fe0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc00185da10) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69840, 0xc000937c20, 0xc0002310c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1467 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc00185dad0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69890, 0xc000937c20, 0xc0002311a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1468 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc00185db00) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d698b0, 0xc000937c20, 0xc000be5080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1469 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc00193c600, 0x4886ea0, 0xc000102580, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc00185db30) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d698c0, 0xc000937c20, 0xc000102600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1470 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc00185db60) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d699b0, 0xc000937c20, 0xc00185d410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1476 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001bcb080) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1472 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc000be5340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 1473 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc00185d560) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69aa0, 0xc000937c20, 0xc0002a5960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1474 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc00185d9e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001d69b40, 0xc000937c20, 0xc0002bc9e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1503 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc000c3c150) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b73b0, 0xc000937c20, 0xc0003fc580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1492 [select]: net.(*pipe).read(0xc001580680, 0xc001e98800, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/pipe.go:164 +0x19b net.(*pipe).Read(0xc001580680, 0xc001e98800, 0x791, 0x791, 0xc001e9880d, 0x11, 0x10) /usr/local/go/src/net/pipe.go:147 +0x4d crypto/tls.(*atLeastReader).Read(0xc004742e40, 0xc001e98800, 0x791, 0x791, 0x2e, 0x482c600, 0xc001463ac0) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001e7b3d8, 0x482c340, 0xc004742e40, 0x6fcab5, 0x3aee1e0, 0x3d5d2c0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001e7b180, 0x7f6d0c22ebd8, 0xc001580680, 0x5, 0xc001580680, 0x11) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001e7b180, 0x40e5100, 0xc001e7b2b8, 0x0) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001e7b180, 0xc0007a0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001c7e540, 0xc0016a02d8, 0x9, 0x9, 0x6fe9a9, 0xc0011d5380, 0xc001463e40) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001c7e540, 0xc0016a02d8, 0x9, 0x9, 0x9, 0x0, 0xc001d78358, 0x74a6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0016a02d8, 0x9, 0x9, 0x482c140, 0xc001c7e540, 0x0, 0xc000000000, 0xc001463ea7, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0016a02a0, 0xc004f37720, 0xc004f37720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0001ba000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1808 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ee70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e60, 0xc000937c20, 0xc000335800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1809 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087eea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e70, 0xc000937c20, 0xc000335820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1601 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001084000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1560 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228bd8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001952518, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001952500, 0xc000f93800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001952500, 0xc000f93800, 0x75b, 0x75b, 0xc001e8b938, 0x701, 0xc001c5bd80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0016683c0, 0xc000f93800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc001675020, 0xc000f93800, 0x75b, 0x75b, 0xc000f9380d, 0x65, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc004a0e9c0, 0xc000f93800, 0x75b, 0x75b, 0x82, 0x482c600, 0xc001e8ba50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001c5be58, 0x482c340, 0xc004a0e9c0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001c5bc00, 0x7f6d0700c008, 0xc001675020, 0x5, 0xc001675020, 0x65) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001c5bc00, 0x40e5100, 0xc001c5bd38, 0xb) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001c5bc00, 0xc001132000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0019b7020, 0xc00111c818, 0x9, 0x9, 0xc002713e88, 0xc00751fb60, 0x5c) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0019b7020, 0xc00111c818, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111c818, 0x9, 0x9, 0x482c140, 0xc0019b7020, 0x0, 0xb00000000, 0xc002700f00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111c7e0, 0xc001d8b170, 0xc001d8b170, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00168b200, 0xc001d8b7d0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc000c12a80, 0x48fef60, 0xc00168b200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc000c12a80, 0x48fef60, 0xc00168b200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1559 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00168b200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1558 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001784880, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0019b7860, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00168b200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1557 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0017848c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0019b7200, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000366a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 1556 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c228ca8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001952418, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001952400, 0xc000f93000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001952400, 0xc000f93000, 0x791, 0x791, 0x203000, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc00075c020, 0xc000f93000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00379d6e0, 0xc000f93000, 0x791, 0x791, 0x6e, 0x482c600, 0xc001e89a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001dfe5d8, 0x482c340, 0xc00379d6e0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001dfe380, 0x48317e0, 0xc00075c020, 0x5, 0xc00075c020, 0xc002b43680) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001dfe380, 0x40e5100, 0xc001dfe4b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001dfe380, 0xc001160000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0019b7140, 0xc00111c8f8, 0x9, 0x9, 0xc0027317f8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0019b7140, 0xc00111c8f8, 0x9, 0x9, 0x9, 0xc006a62960, 0x485e320, 0xc00216fe60) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111c8f8, 0x9, 0x9, 0x482c140, 0xc0019b7140, 0x0, 0xc000000000, 0xc00449ad00, 0x2b43600) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111c8c0, 0xc002b43650, 0xc002b43650, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000366a80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 1807 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ee40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e50, 0xc000937c20, 0xc0003357e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1638 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c85) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363530, 0xc000937c20, 0xc0003634e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1637 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf77a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf77a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003634d0, 0xc000937c20, 0xc000363490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1636 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c76) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363480, 0xc000937c20, 0xc000363420) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1635 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363410, 0xc000937c20, 0xc0003633c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1643 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c89) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7980) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003637a0, 0xc000937c20, 0xc0003636f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1644 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c95) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf79b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf79b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003637f0, 0xc000937c20, 0xc0003637b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1645 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c21) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf79e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf79e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363840, 0xc000937c20, 0xc000363800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1646 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363890, 0xc000937c20, 0xc000363850) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1647 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c92) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003638e0, 0xc000937c20, 0xc0003638a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1648 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c25) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363930, 0xc000937c20, 0xc0003638f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1649 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c94) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363980, 0xc000937c20, 0xc000363940) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1650 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003639d0, 0xc000937c20, 0xc000363990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1651 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c32) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363a30, 0xc000937c20, 0xc0003639e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1652 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ab0, 0xc000937c20, 0xc000363a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1653 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c9b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b20, 0xc000937c20, 0xc000363ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1654 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c91) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b60, 0xc000937c20, 0xc000363b30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1655 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c31) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363b90, 0xc000937c20, 0xc000363b80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1656 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c20) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c20, 0xc000937c20, 0xc000363c10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1657 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c23) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c40, 0xc000937c20, 0xc000363c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1658 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c60, 0xc000937c20, 0xc000363c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1659 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363c90, 0xc000937c20, 0xc000363c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1660 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c97) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7cb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363cc0, 0xc000937c20, 0xc000363cb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1661 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c47) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7ce0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ce0, 0xc000937c20, 0xc000363cd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1662 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c22) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7d10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d00, 0xc000937c20, 0xc000363cf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1663 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c38) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d20, 0xc000937c20, 0xc000363d10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1664 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d40, 0xc000937c20, 0xc000363d30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1665 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c48) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7da0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d60, 0xc000937c20, 0xc000363d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1666 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c54) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7dd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363d80, 0xc000937c20, 0xc000363d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1667 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c39) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363da0, 0xc000937c20, 0xc000363d90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1668 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c55) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7e30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363dd0, 0xc000937c20, 0xc000363dc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1669 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c30) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7e60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363df0, 0xc000937c20, 0xc000363de0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1670 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c43) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e20, 0xc000937c20, 0xc000363e10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1671 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e50, 0xc000937c20, 0xc000363e30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1672 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c37) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7ef0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363e80, 0xc000937c20, 0xc000363e60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1673 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7f20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363eb0, 0xc000937c20, 0xc000363e90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1674 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ef0, 0xc000937c20, 0xc000363ee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1675 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c63) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363f60, 0xc000937c20, 0xc000363f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1676 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c51) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363f90, 0xc000937c20, 0xc000363f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1677 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c57) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363fc0, 0xc000937c20, 0xc000363fb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1678 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c65) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e030) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000363ff0, 0xc000937c20, 0xc000363fe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1679 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248670, 0xc000937c20, 0xc0002484e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1680 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c62) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e090) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002486a0, 0xc000937c20, 0xc000248690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1681 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e0c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e0c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002486c0, 0xc000937c20, 0xc0002486b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1682 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c70) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e0f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e0f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248710, 0xc000937c20, 0xc000248700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1683 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c60) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248790, 0xc000937c20, 0xc000248770) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1684 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c52) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e150) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002487b0, 0xc000937c20, 0xc0002487a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1685 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e180) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002487d0, 0xc000937c20, 0xc0002487c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1686 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c66) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e1b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e1b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248810, 0xc000937c20, 0xc000248800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1687 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e1e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e1e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248830, 0xc000937c20, 0xc000248820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1688 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e2a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e2a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248850, 0xc000937c20, 0xc000248840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1689 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002488a0, 0xc000937c20, 0xc000248860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1690 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e3c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002488d0, 0xc000937c20, 0xc0002488b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1691 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c69) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e3f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e3f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002488f0, 0xc000937c20, 0xc0002488e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1692 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248920, 0xc000937c20, 0xc000248910) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1693 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c6b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248950, 0xc000937c20, 0xc000248940) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1694 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c72) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e4b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e4b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248980, 0xc000937c20, 0xc000248970) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1695 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c73) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e4e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e4e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002489b0, 0xc000937c20, 0xc000248990) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1696 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002489e0, 0xc000937c20, 0xc0002489c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1697 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c80) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e570) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248a50, 0xc000937c20, 0xc0002489f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1698 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e5a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e5a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248a70, 0xc000937c20, 0xc000248a60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1699 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c77) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e5d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248a90, 0xc000937c20, 0xc000248a80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1700 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248ab0, 0xc000937c20, 0xc000248aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1701 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c82) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e630) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248ad0, 0xc000937c20, 0xc000248ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1702 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c81) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e690) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248b00, 0xc000937c20, 0xc000248af0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1703 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc00087e6f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc00087e6f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248b30, 0xc000937c20, 0xc000248b10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1704 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c79) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc001bf7530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001bf7530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248b60, 0xc000937c20, 0xc000248b40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1705 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c7b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248b80, 0xc000937c20, 0xc000248b70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1706 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c9a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248bb0, 0xc000937c20, 0xc000248ba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1707 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c96) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248c20, 0xc000937c20, 0xc000248c10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1708 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c86) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834cf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248c40, 0xc000937c20, 0xc000248c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1709 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c90) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248c60, 0xc000937c20, 0xc000248c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1710 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248c80, 0xc000937c20, 0xc000248c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1711 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c98) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248cb0, 0xc000937c20, 0xc000248ca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1712 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c24) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248cd0, 0xc000937c20, 0xc000248cc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1713 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c9c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248d40, 0xc000937c20, 0xc000248ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1714 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000834ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000834ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248d70, 0xc000937c20, 0xc000248d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1715 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c1f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248da0, 0xc000937c20, 0xc000248d80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1716 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c93) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835050) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835050) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248dd0, 0xc000937c20, 0xc000248db0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1717 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248df0, 0xc000937c20, 0xc000248de0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1718 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c1e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008350b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008350b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248e20, 0xc000937c20, 0xc000248e10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1719 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c26) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008350e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008350e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248e50, 0xc000937c20, 0xc000248e40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1720 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c2e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248e80, 0xc000937c20, 0xc000248e70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1721 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c36) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248f10, 0xc000937c20, 0xc000248f00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1722 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c28) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248f30, 0xc000937c20, 0xc000248f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1723 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c27) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008351a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008351a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248f50, 0xc000937c20, 0xc000248f40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1724 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008351d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008351d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248f70, 0xc000937c20, 0xc000248f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1725 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c29) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248f90, 0xc000937c20, 0xc000248f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1726 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c41) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248fb0, 0xc000937c20, 0xc000248fa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1727 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c34) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248fd0, 0xc000937c20, 0xc000248fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1728 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c33) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248ff0, 0xc000937c20, 0xc000248fe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1729 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c40) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008352c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008352c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249010, 0xc000937c20, 0xc000249000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1730 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c35) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008353b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008353b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249030, 0xc000937c20, 0xc000249020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1731 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c46) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008354d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008354d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249060, 0xc000937c20, 0xc000249050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1732 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249090, 0xc000937c20, 0xc000249070) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1733 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002490c0, 0xc000937c20, 0xc0002490a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1734 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c42) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002490e0, 0xc000937c20, 0xc0002490d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1735 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c49) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008355f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008355f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249130, 0xc000937c20, 0xc000249120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1736 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002491a0, 0xc000937c20, 0xc000249170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1737 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002491d0, 0xc000937c20, 0xc0002491b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1738 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c45) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002491f0, 0xc000937c20, 0xc0002491e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1739 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c58) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008356b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008356b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249240, 0xc000937c20, 0xc000249210) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1740 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c56) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008356e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008356e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249270, 0xc000937c20, 0xc000249250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1741 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c3c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002492b0, 0xc000937c20, 0xc000249280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1742 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249320, 0xc000937c20, 0xc000249310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1743 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c59) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249450, 0xc000937c20, 0xc0002493b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1744 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c44) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008357a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008357a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249490, 0xc000937c20, 0xc000249460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1745 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c50) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008357d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008357d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249530, 0xc000937c20, 0xc0002494b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1746 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c53) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835800) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249560, 0xc000937c20, 0xc000249540) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1747 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008358c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008358c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002495e0, 0xc000937c20, 0xc0002495b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1748 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c61) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008359b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008359b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249650, 0xc000937c20, 0xc0002495f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1749 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c4d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc0008359e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008359e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002498b0, 0xc000937c20, 0xc000249720) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1750 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c5a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249910, 0xc000937c20, 0xc0002498c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1751 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c67) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002499a0, 0xc000937c20, 0xc000249920) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1752 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c71) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835a70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002499c0, 0xc000937c20, 0xc0002499b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1753 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c64) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002499f0, 0xc000937c20, 0xc0002499e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1754 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c68) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249a10, 0xc000937c20, 0xc000249a00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1755 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c75) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249a30, 0xc000937c20, 0xc000249a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1756 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c78) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249a50, 0xc000937c20, 0xc000249a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1757 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c8b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249a70, 0xc000937c20, 0xc000249a60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1758 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00157a250, 0x3c74) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00157a240) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001952780, 0x489e1a0, 0xc000835bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc000835bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249a90, 0xc000937c20, 0xc000249a80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1759 [semacquire]: sync.runtime_Semacquire(0xc0019527f8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc0019527f0) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249ab0, 0xc000937c20, 0xc000249aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1760 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc000932000, 0x489e1a0, 0xc000835c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249ad0, 0xc000937c20, 0xc000249ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1761 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc000932000, 0x489e1a0, 0xc00087e7e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249af0, 0xc000937c20, 0xc000249ae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1762 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc00087e8d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249b40, 0xc000937c20, 0xc0003b2400) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1763 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc00087e9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249b50, 0xc000937c20, 0xc000bef770) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1764 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc00087e9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249b80, 0xc000937c20, 0xc000bef900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1765 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc00087ea20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249b90, 0xc000937c20, 0xc000bef950) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1497 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startWriteNodeStatus.func1(0x489e1a0, 0xc000969fb0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:781 +0x101 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b7150, 0xc000937c20, 0xc001dc59e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1767 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc00087eab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249bd0, 0xc000937c20, 0xc000249bc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1768 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001bdce50, 0x9c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001bdce40) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007844d0, 0x489e1a0, 0xc0005ac9f0, 0xc000fd1740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc0005ac9f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e1a0, 0xc0005ac9f0, 0xc001420e40, 0x25, 0x0, 0x0, 0xc0003b2420) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 1769 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc00087eb10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249be0, 0xc000937c20, 0xc0003b2440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1770 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc00087eb70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249bf0, 0xc000937c20, 0xc0003b2460) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1771 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc00087e720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249c00, 0xc000937c20, 0xc0003b2480) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1772 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc000835c80) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249d90, 0xc000937c20, 0xc0003b2640) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1773 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc000835ce0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249e90, 0xc000937c20, 0xc0003b2680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1774 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc000f87b80, 0xc0017fe4d0, 0xc000249f00, 0x40e5438, 0xc0005eaae0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc000835f80) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249f10, 0xc000937c20, 0xc0005add70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1775 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc000835fb0) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249f50, 0xc000937c20, 0xc0003b2700) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1776 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc0008b8210) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003100b0, 0xc000937c20, 0xc0005addd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1777 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc0008b8270) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000310110, 0xc000937c20, 0xc0003100f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1778 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc00087f890) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000310150, 0xc000937c20, 0xc0003b27c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1779 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc000a00b70) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000310230, 0xc000937c20, 0xc0003101c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1780 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc00087f830) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000310240, 0xc000937c20, 0xc0003b2820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1781 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc000a00bd0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000310250, 0xc000937c20, 0xc0003b2860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1782 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc000a00c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003103f0, 0xc000937c20, 0xc000befb30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1806 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ede0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e40, 0xc000937c20, 0xc0003357c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1805 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ecc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e30, 0xc000937c20, 0xc0003357a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1804 [chan send]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ebd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:767 +0x1dc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e10, 0xc000937c20, 0xc000335780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1803 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087eba0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e00, 0xc000937c20, 0xc000335760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1811 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc00087ef00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343e90, 0xc000937c20, 0xc000335860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1812 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc001af5ba0, 0x489e1a0, 0xc00087ea80, 0xc000937c20, 0x299cfa23, 0xed500c077, 0x0, 0xc003a90000, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:217 +0x1d1 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc003a90000, 0xc0005ac401) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc0005ac4e0, 0xc004d9a940) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc00087ea80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000343ea0, 0xc000937c20, 0xc000335880) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2802 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc001915e00, 0x7f6d05d41348, 0xc001842040, 0x7f6d05d41348, 0xc001842040) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc001915e00, 0x48fa760, 0xc001842040, 0xc02f00010301, 0xc001842040) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc0009cf800, 0x0, 0xc0026126c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc0009cf800, 0xc0030749e0, 0x40e1e60, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc0009cf800, 0xc0030749e0, 0x40e1e60, 0x489e1a0, 0xc00333dc80) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc000f85900, 0xc00111bb30, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc000f85900, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004492f0, 0xc00193c600, 0x48fef60, 0xc00260a480, 0xc000f85900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2584 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001a36c60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a98c0, 0xc000a30090, 0xc0036284a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2381 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000c22000, 0xc001de7e60, 0xc001ae6320, 0xf, 0xc0028f04e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc00079ff80) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001dbdc30, 0xc000a30090, 0xc0009f4230) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1482 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc000c22000, 0xc0014c22a0, 0xc0003aa520, 0xf, 0xc0015e80c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc0015b2090) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0017f4280, 0xc000a30090, 0xc0018da690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1484 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000632840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 1633 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc00293e000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2621 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc00171e720) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffbc0, 0xc000937c20, 0xc0019e04e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1870 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3648, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0016bea98, 0x72, 0x7900, 0x79a3, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0016bea80, 0xc00280c000, 0x79a3, 0x79a3, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0016bea80, 0xc00280c000, 0x79a3, 0x79a3, 0xc00129e938, 0x7901, 0xc001564500) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f941f0, 0xc00280c000, 0x79a3, 0x79a3, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002259320, 0xc00280c000, 0x79a3, 0x79a3, 0xc00280c00d, 0x59, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc001dc45e0, 0xc00280c000, 0x79a3, 0x79a3, 0x76, 0x482c600, 0xc00129ea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0015645d8, 0x482c340, 0xc001dc45e0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001564380, 0x7f6d0700c008, 0xc002259320, 0x5, 0xc002259320, 0x59) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001564380, 0x40e5100, 0xc0015644b8, 0x7) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001564380, 0xc0027f4000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc0018948a0, 0xc0003e1b58, 0x9, 0x9, 0xc0010608c8, 0xc002932fa0, 0x50) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc0018948a0, 0xc0003e1b58, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003e1b58, 0x9, 0x9, 0x482c140, 0xc0018948a0, 0x0, 0x700000000, 0xc002d1e000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e1b20, 0xc0015efad0, 0xc0015efad0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001892180, 0xc00162cb70, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc001892180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc001892180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 1866 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc00162fbd0, 0x1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc00162fbc0) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/sql.(*StmtBuf).CurCmd(0xc0015eaf20, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_io.go:437 +0x89 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc0028f4000, 0x489e0e0, 0xc00162fc00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1191 +0x52 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc0028f4000, 0x489e0e0, 0xc00162fc00, 0xc001a5d0c0, 0x5400, 0x15000, 0xc001a5d158, 0xc001898be0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn(0xc0006a2a00, 0x489e0e0, 0xc00162fc00, 0xc0028f4000, 0x5400, 0x15000, 0xc001a5d158, 0xc001898be0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:444 +0xce github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1(0xc000318a71, 0xc00180e3a0, 0x489e0e0, 0xc00162fc00, 0xc001898be0, 0xc0006a2a00, 0xc0015ea900, 0x48c0ea0, 0xc00180e380, 0xc001b07b00, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:584 +0x21c created by github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:513 +0x17b goroutine 2620 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001dff880, 0xc000d54000, 0x489e1a0, 0xc00171e4e0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2120 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0018a82d0, 0x48fa580, 0xc0016b64f0, 0x485c120, 0xc0018a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00341a000, 0x0, 0xc002e646c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00341a000, 0xc0003cc8e0, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00341a000, 0xc0003cc8e0, 0x40e1ba8, 0x489e1a0, 0xc0015eed50) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc003110000, 0xc000b24e00, 0xc0009a98f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc003110000, 0xc000b24e00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003abbd0, 0xc00262b200, 0x48fef60, 0xc003110000, 0xc000b24e00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2119 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0015e77c0, 0xc0003cc6f0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0015e77c0, 0xc0003cc6f0, 0x5, 0x5, 0xc11b10, 0xc0005e01c0, 0xc004c1f978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0015eeab0, 0xc0003cc6f0, 0x5, 0x5, 0x74cff0, 0xc004c1f9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0015eeab0, 0xc0003cc6f0, 0x5, 0x5, 0x5, 0xf, 0xc000079c80, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000b24d00, 0xc0003cc6f0, 0x5, 0x5, 0x83e500, 0xc005d4a8c0, 0xc004c1fa78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003cc6e0, 0x7fffffff, 0x3c06120, 0xc000079c80, 0xc004c1fa0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003cc6e0, 0xc000b24d00, 0x0, 0x0, 0x7fffffff, 0xc004c1fcb8, 0x4886820, 0x72b56a8, 0x0, 0xc00755b160, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003cc6e0, 0x7f6d0c1990d0, 0xc000292db0, 0xc000b24d00, 0x0, 0x0, 0x3de8800, 0xc00694f040, 0x7fffffff, 0xc004c1fcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001c6a280, 0x3de8800, 0xc00694f040, 0xc004c1fcb8, 0xc0059d0f50, 0xc00694f040) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001c6a280, 0xc00007e070, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001b1ad80, 0xc004c1fe00, 0xc004c1fdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001b1ad80, 0x3de8800, 0xc00694f040, 0x2c5e8e0, 0xc000a15500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0016b6350, 0x489e1a0, 0xc0015eeb40, 0xc006a0d9c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0016b6350, 0xc0015f8080, 0xc0002a8960, 0x489e1a0, 0xc0015eeb40, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0015eeb40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b6370, 0xc000a30090, 0xc0015eeae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2690 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0018aea40, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc001c7fc80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0014f0780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 1151 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0018a82d0, 0x1, 0xc002e65320, 0xc001db8b80, 0x48fa520, 0xc0015a9430, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001799650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a9390, 0xc0019e4510, 0xc0017995f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2118 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0009f2e00, 0xc001b1ad80, 0x489e1a0, 0xc0015ee990) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2117 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0002a8960, 0x3, 0xc002e20b40, 0xc0015f8080, 0x48fa520, 0xc0016b6350, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0015ee8a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b6260, 0xc000a30090, 0xc0015ee840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2114 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc001a94410, 0xc0002c91f0, 0x5, 0x5, 0x4fe4, 0xc005ec0b40, 0x9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc001a94410, 0xc0002c91f0, 0x5, 0x5, 0xc11b10, 0xc00141e720, 0xc001bc7970) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00195f410, 0xc0002c91f0, 0x5, 0x5, 0x0, 0xc001bc7998, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00195f410, 0xc0002c91f0, 0x5, 0x5, 0x5, 0x14dfbb6, 0xc005ec0b01, 0xc001bc7a50) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002165f00, 0xc0002c91f0, 0x5, 0x5, 0xc00075c340, 0x0, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002c91e0, 0x7fffffff, 0xc0077a2be0, 0x3, 0x4, 0x3, 0x4, 0x9) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002c91e0, 0xc002165f00, 0x0, 0x0, 0x7fffffff, 0xc001bc7cb0, 0x4886820, 0x72b56a8, 0x0, 0xa, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002c91e0, 0x7f6d0c1990d0, 0xc000292db0, 0xc002165f00, 0x0, 0x0, 0x3dd1ee0, 0xc001a34af0, 0x7fffffff, 0xc001bc7cb0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001580280, 0x3dd1ee0, 0xc001a34af0, 0xc001bc7cb0, 0xc001bc7d30, 0xc001a34af0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001580280, 0xc001bc7d40, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001c73680, 0xc001bc7df8, 0xc001bc7de8, 0x0, 0x48) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001c73680, 0x3dd1ee0, 0xc001a34af0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/gossip.(*gossipGossipClient).Recv(0xc0010ffde0, 0x489e1a0, 0xc00195f4a0, 0xc00262b380) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:553 +0x62 github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2.1(0xc0011f4e40, 0x48f9ec0, 0xc0010ffde0, 0xc00196fee0, 0x489e1a0, 0xc00195f4a0, 0xc00262b380, 0x30, 0x3d2f600) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:322 +0x4a github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip.func2(0x489e1a0, 0xc00195f4a0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:337 +0xde github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffe70, 0xc0019e4510, 0xc001743ec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2121 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0015e7b30, 0xc0003cc8d0, 0x5, 0x5, 0xc0055fbe00, 0x1f0, 0xc004d90000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0015e7b30, 0xc0003cc8d0, 0x5, 0x5, 0xc11b10, 0xc0005e0220, 0xc0031ffaf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0015eecf0, 0xc0003cc8d0, 0x5, 0x5, 0x72, 0xc0031ffb20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0015eecf0, 0xc0003cc8d0, 0x5, 0x5, 0x5, 0xc0031ffbd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000b24e00, 0xc0003cc8d0, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0003cc8c0, 0x7fffffff, 0x30, 0xc00456a7e0, 0x4, 0xc0031ffc70, 0xc001c5a000, 0x7f6d0c276f80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0003cc8c0, 0xc000b24e00, 0x0, 0x0, 0x7fffffff, 0xc0031ffe30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0003cc8c0, 0x7f6d0c1990d0, 0xc000292db0, 0xc000b24e00, 0x0, 0x0, 0x3de86e0, 0xc0036ad2e0, 0x7fffffff, 0xc0031ffe30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00341a000, 0x3de86e0, 0xc0036ad2e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0016b64f0, 0x489e1a0, 0xc0015eedb0, 0xc004d901f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0016b64f0, 0xc0018a82d0, 0x489e1a0, 0xc0015eedb0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0015eedb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016b6510, 0xc0019e4510, 0xc0015eed80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1998 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b98c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56600, 0xc0019e4510, 0xc00039ece0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1999 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b98f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56610, 0xc0019e4510, 0xc00039ed00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2001 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).waitAndProcess(0xc00215c4e0, 0x489e1a0, 0xc0008b9200, 0xc0019e4510, 0x6dd3e81, 0xed500c078, 0x0, 0xc00459a000, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:217 +0x1d1 github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1.1(0xc00459a000, 0xc00195fb01) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:279 +0x81 github.com/cockroachdb/cockroach/pkg/storage.(*storeReplicaVisitor).Visit(0xc00195fb90, 0xc006e52840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:350 +0x18d github.com/cockroachdb/cockroach/pkg/storage.(*replicaScanner).scanLoop.func1(0x489e1a0, 0xc0008b9200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scanner.go:277 +0x19b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56640, 0xc0019e4510, 0xc00039eda0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2572 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0018ae390, 0x9e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0018ae380) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0005c8230, 0x489e1a0, 0xc001a9b530, 0xc003a02060) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc001a9b530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc001a9b530, 0xc0032cf040, 0x1d, 0x0, 0x0, 0xc001b35f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2169 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001c10900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2130 [semacquire]: sync.runtime_SemacquireMutex(0xc00028d334, 0x7f6d0c276800) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc00028d330) /usr/local/go/src/sync/mutex.go:134 +0x109 sync.(*Once).Do(0xc00028d330, 0xc0023f3f50) /usr/local/go/src/sync/once.go:40 +0x3b github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc001d8ab40) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016ee380, 0xc0019e4510, 0xc0002bda60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2026 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606c90) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003116b0, 0xc0019e4510, 0xc0003116a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2027 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606c60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003116f0, 0xc0019e4510, 0xc0003116c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2028 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000724c90) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003117d0, 0xc0019e4510, 0xc0003117c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2029 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc000724c60) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0003117f0, 0xc0019e4510, 0xc0003117e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2030 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606c00) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311810, 0xc0019e4510, 0xc000311800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2031 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606bd0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311830, 0xc0019e4510, 0xc000311820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2032 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc001606c30) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311860, 0xc0019e4510, 0xc000311840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2033 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).initRunners.func1(0x489e1a0, 0xc00079ee10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:94 +0x140 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311880, 0xc0019e4510, 0xc000311870) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2034 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc001b1a480, 0x489e0e0, 0xc00172a540) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 2035 [select]: github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).gcDaemon(0xc001b1a5a0, 0x489e0e0, 0xc00172a540) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1286 +0xdf created by github.com/cockroachdb/cockroach/pkg/util/log.NewSecondaryLogger /go/src/github.com/cockroachdb/cockroach/pkg/util/log/secondary_log.go:79 +0x2a7 goroutine 2036 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).RefreshLeases.func1(0x489e1a0, 0xc000724cc0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1716 +0x370 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311e40, 0xc0019e4510, 0xc000be6a50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2037 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*LeaseManager).PeriodicallyRefreshSomeLeases.func1(0x489e1a0, 0xc00079eea0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:1797 +0x257 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311e60, 0xc0019e4510, 0xc000311e50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2038 [select]: github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps.func1(0x40e5438, 0xc001845a90, 0xc0017fe2a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:181 +0x11b created by github.com/cockroachdb/cockroach/pkg/util/hlc.(*Clock).StartMonitoringForwardClockJumps /go/src/github.com/cockroachdb/cockroach/pkg/util/hlc/hlc.go:174 +0x105 goroutine 2039 [chan receive]: github.com/cockroachdb/cockroach/pkg/util/netutil.MakeServer.func2(0x489e1a0, 0xc000724d80) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:107 +0x65 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311f00, 0xc0019e4510, 0xc001d2eb00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2040 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func1(0x489e1a0, 0xc00079eed0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1751 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311f40, 0xc0019e4510, 0xc001d2eb40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2041 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc3578, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a398, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000f7a380, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000f7a380, 0xc000496000, 0x7f6d0c274d98, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000f94710, 0x71f47f, 0xc000f91b30, 0xc0020e0ef0) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000f94710, 0x40e2688, 0xc00172a6c0, 0xc000267894, 0x0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc00172a6c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func2(0x489e1a0, 0xc000724db0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1764 +0x2f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311f80, 0xc0019e4510, 0xc001d2eba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2042 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 net/http.(*Server).Serve(0xc001af5e10, 0x4886ea0, 0xc001d2eb60, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func3(0x489e1a0, 0xc001606cc0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1780 +0x127 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311f90, 0xc0019e4510, 0xc000be6c90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2043 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 crypto/tls.(*listener).Accept(0xc001d2ebc0, 0x3d2f600, 0xc00158d7a0, 0x3a88520, 0x6f95560) /usr/local/go/src/crypto/tls/tls.go:62 +0x37 net/http.(*Server).Serve(0xc00196fa00, 0x486e6e0, 0xc001d2ebc0, 0x0, 0x0) /usr/local/go/src/net/http/server.go:2859 +0x22d github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeUI.func4(0x489e1a0, 0xc00158d710) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1791 +0x3f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000311fb0, 0xc0019e4510, 0xc001d2ebe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2044 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func1(0x489e1a0, 0xc00158d7d0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1654 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002486a0, 0xc0019e4510, 0xc001d2ec40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2045 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func3(0x489e1a0, 0xc00158d800) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1706 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002486c0, 0xc0019e4510, 0xc00172a740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2046 [chan receive]: github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.muxListener.Accept(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:184 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).Serve(0xc00262b200, 0x4886ea0, 0xc001d2ec80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:587 +0x1e9 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func4(0x489e1a0, 0xc000724de0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1721 +0x46 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248700, 0xc0019e4510, 0xc001d2eca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2047 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func1(0x489e1a0, 0xc001607230) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1198 +0x7d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002488a0, 0xc0019e4510, 0xc000a00120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1994 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1(0x489e1a0, 0xc0008b96b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:748 +0x15e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d56550, 0xc0019e4510, 0xc00039ebc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2049 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00172aa40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2050 [chan receive]: github.com/cockroachdb/cockroach/pkg/server.(*Server).Start.func4(0x489e1a0, 0xc00158d860) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1228 +0x54 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002489f0, 0xc0019e4510, 0xc001d2efa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2051 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001892d20, 0xc000001cbb) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001892d10) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).syncLoop(0xc001892c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:674 +0x70 created by github.com/cockroachdb/cockroach/pkg/storage/engine.(*RocksDB).open /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:661 +0x37c goroutine 2052 [select]: github.com/cockroachdb/cockroach/pkg/server.startAssertEngineHealth.func1(0x489e1a0, 0xc000a003c0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_engine_health.go:44 +0x16b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248b60, 0xc0019e4510, 0xc000a00330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2053 [chan receive]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).start.func3(0x489e1a0, 0xc000a003f0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:383 +0x5b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248c80, 0xc0019e4510, 0xc001d2f8e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2054 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).bootstrap.func1(0x489e1a0, 0xc000a00450) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1365 +0x3b4 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248cb0, 0xc0019e4510, 0xc000248ca0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2055 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*Gossip).manage.func1(0x489e1a0, 0xc000a007e0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.go:1400 +0x33c github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248cd0, 0xc0019e4510, 0xc000248cc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2056 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc34a8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a418, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000f7a400, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000f7a400, 0xc000110b01, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000f94728, 0xc0018d3e80, 0xc0018d3e88, 0x28) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000f94728, 0x40e2690, 0xc00172a700, 0x48faa00, 0xc000110b48) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*cMux).Serve(0xc00172a700, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:124 +0x88 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1.1() /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1731 +0x2f sync.(*Once).Do(0xc00028d330, 0xc0020e0750) /usr/local/go/src/sync/once.go:44 +0xb3 github.com/cockroachdb/cockroach/pkg/server.(*Server).startListenRPCAndSQL.func5.1(0x489e1a0, 0xc000a00390) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1730 +0x63 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248ce0, 0xc0019e4510, 0xc001d2f900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2057 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*client).gossip(0xc00196fee0, 0x489e1a0, 0xc000a005a0, 0xc00262b380, 0x48f9ec0, 0xc0010ffde0, 0xc0019e4510, 0xc00028d670, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:366 +0x344 github.com/cockroachdb/cockroach/pkg/gossip.(*client).startLocked.func1(0x489e1a0, 0xc000a005a0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/client.go:125 +0x3c2 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000248e10, 0xc0019e4510, 0xc00172ab00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2058 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00172ae40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2059 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc001b176e0, 0xc001ae65b0, 0xf, 0xc001664060, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc000a007b0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000249010, 0xc0019e4510, 0xc001747c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2060 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001912840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2017 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc00293e380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 1984 [select]: github.com/cockroachdb/cockroach/pkg/gossip.(*server).Gossip(0xc000c26200, 0x48f9f20, 0xc000d566e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/server.go:193 +0x625 github.com/cockroachdb/cockroach/pkg/gossip._Gossip_Gossip_Handler(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e000, 0x0, 0xc001e21e00) /go/src/github.com/cockroachdb/cockroach/pkg/gossip/gossip.pb.go:569 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e000, 0xc001d6b080, 0x40df160, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3e992e0, 0xc000c26200, 0x48eb940, 0xc001d4e000, 0xc001d6b080, 0x40df160, 0x489e1a0, 0xc00079fda0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc0021c2000, 0xc000b24500, 0xc000be7800, 0x6f9bba0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc0021c2000, 0xc000b24500, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0005e98d0, 0xc000c12a80, 0x48fef60, 0xc0021c2000, 0xc000b24500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2175 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).run(0xc00174c460, 0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:411 +0x36f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e120, 0xc00010c010, 0xc00192a210, 0x22, 0x0, 0x0, 0xc0010fffc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2176 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func1(0x489e1a0, 0xc001574d20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:151 +0x4a github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffff0, 0xc0019e4510, 0xc0002c92a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2177 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4193) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898010, 0xc0019e4510, 0xc001898000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2210 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4172) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898030, 0xc0019e4510, 0xc001898020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2211 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574db0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898050, 0xc0019e4510, 0xc001898040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2212 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4184) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898070, 0xc0019e4510, 0xc001898060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2213 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4183) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574e10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574e10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018980a0, 0xc0019e4510, 0xc001898080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2214 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4182) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018980d0, 0xc0019e4510, 0xc0018980b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2215 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4181) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574e70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018980f0, 0xc0019e4510, 0xc0018980e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2216 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898110, 0xc0019e4510, 0xc001898100) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2217 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4185) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898130, 0xc0019e4510, 0xc001898120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2218 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898150, 0xc0019e4510, 0xc001898140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2219 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898170, 0xc0019e4510, 0xc001898160) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2220 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4189) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574f60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574f60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898190, 0xc0019e4510, 0xc001898180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2221 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4130) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574f90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574f90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018981b0, 0xc0019e4510, 0xc0018981a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2222 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574fc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574fc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018981f0, 0xc0019e4510, 0xc0018981c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2223 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898210, 0xc0019e4510, 0xc001898200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2224 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4197) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575020) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898230, 0xc0019e4510, 0xc001898220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2225 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575080) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898250, 0xc0019e4510, 0xc001898240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2226 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4190) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015750b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015750b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898270, 0xc0019e4510, 0xc001898260) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2227 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015750e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015750e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898290, 0xc0019e4510, 0xc001898280) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2228 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018982b0, 0xc0019e4510, 0xc0018982a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2229 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a5) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018982d0, 0xc0019e4510, 0xc0018982c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2230 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018982f0, 0xc0019e4510, 0xc0018982e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2231 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4194) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015751a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015751a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898340, 0xc0019e4510, 0xc001898300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2232 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a8) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015751d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015751d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898370, 0xc0019e4510, 0xc001898350) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2233 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a0) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898390, 0xc0019e4510, 0xc001898380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2234 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4191) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018983c0, 0xc0019e4510, 0xc0018983b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2235 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898400, 0xc0019e4510, 0xc0018983e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2236 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a1) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898420, 0xc0019e4510, 0xc001898410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2237 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4199) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015752c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015752c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898440, 0xc0019e4510, 0xc001898430) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2238 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015752f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015752f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898460, 0xc0019e4510, 0xc001898450) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2239 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x418c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575320) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898480, 0xc0019e4510, 0xc001898470) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2240 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4195) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018984a0, 0xc0019e4510, 0xc001898490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2241 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a3) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018984c0, 0xc0019e4510, 0xc0018984b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2242 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a2) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015753b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015753b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018984e0, 0xc0019e4510, 0xc0018984d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2243 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015753e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015753e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898500, 0xc0019e4510, 0xc0018984f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2244 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x419b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575410) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898520, 0xc0019e4510, 0xc001898510) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2245 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a9) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898540, 0xc0019e4510, 0xc001898530) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2246 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a7) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898560, 0xc0019e4510, 0xc001898550) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2247 [semacquire]: sync.runtime_SemacquireMutex(0xc0077a5894, 0x489e100) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0xc0077a5890) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).tick(0xc0077a5800, 0xc003561a70, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:852 +0x10b github.com/cockroachdb/cockroach/pkg/storage.(*Store).processTick(0xc00084c700, 0x489e1a0, 0xc0015754a0, 0xc3, 0xc001183e01) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3715 +0xf3 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015754a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:205 +0x290 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015754a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898580, 0xc0019e4510, 0xc001898570) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2248 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4196) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015754d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015754d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018985a0, 0xc0019e4510, 0xc001898590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2249 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41ae) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018985c0, 0xc0019e4510, 0xc0018985b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2250 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a4) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018985e0, 0xc0019e4510, 0xc0018985d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2251 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4177) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001574cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001574cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898600, 0xc0019e4510, 0xc0018985f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2252 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4198) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575590) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898620, 0xc0019e4510, 0xc001898610) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2253 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4139) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015755c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015755c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898650, 0xc0019e4510, 0xc001898630) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2254 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015755f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015755f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898670, 0xc0019e4510, 0xc001898660) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2255 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4138) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575620) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018986a0, 0xc0019e4510, 0xc001898690) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2256 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41aa) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575650) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018986d0, 0xc0019e4510, 0xc0018986c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2257 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41ac) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898700, 0xc0019e4510, 0xc0018986e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2258 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015756e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015756e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898750, 0xc0019e4510, 0xc001898710) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2259 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4131) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575710) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898770, 0xc0019e4510, 0xc001898760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2260 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575740) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898790, 0xc0019e4510, 0xc001898780) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2261 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4140) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018987b0, 0xc0019e4510, 0xc0018987a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2262 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4137) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015757a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015757a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018987d0, 0xc0019e4510, 0xc0018987c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2263 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4142) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015757d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015757d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018987f0, 0xc0019e4510, 0xc0018987e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2264 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41ab) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898810, 0xc0019e4510, 0xc001898800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2265 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4150) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575860) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898830, 0xc0019e4510, 0xc001898820) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2266 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015758c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015758c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898850, 0xc0019e4510, 0xc001898840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2267 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4132) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015758f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015758f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898870, 0xc0019e4510, 0xc001898860) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2268 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4145) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575920) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898890, 0xc0019e4510, 0xc001898880) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2269 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4146) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015759b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015759b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018988b0, 0xc0019e4510, 0xc0018988a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2270 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4133) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0015759e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0015759e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018988d0, 0xc0019e4510, 0xc0018988c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2271 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41ad) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575a10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018988f0, 0xc0019e4510, 0xc0018988e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2272 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4135) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898910, 0xc0019e4510, 0xc001898900) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2273 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4136) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898930, 0xc0019e4510, 0xc001898920) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2274 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898950, 0xc0019e4510, 0xc001898940) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2275 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575b30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898970, 0xc0019e4510, 0xc001898960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2276 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898990, 0xc0019e4510, 0xc001898980) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2277 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4148) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575bf0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018989b0, 0xc0019e4510, 0xc0018989a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2278 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4134) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575c20) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018989d0, 0xc0019e4510, 0xc0018989c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2279 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4147) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575c50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018989f0, 0xc0019e4510, 0xc0018989e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2280 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4163) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575c80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898a10, 0xc0019e4510, 0xc001898a00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2281 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898a30, 0xc0019e4510, 0xc001898a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2282 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4157) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898a50, 0xc0019e4510, 0xc001898a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2283 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x413e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575e90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898a80, 0xc0019e4510, 0xc001898a70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2284 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575ec0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ab0, 0xc0019e4510, 0xc001898a90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2285 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4144) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575f50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ae0, 0xc0019e4510, 0xc001898ac0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2286 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4152) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575f80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898b00, 0xc0019e4510, 0xc001898af0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2287 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4155) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575fb0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898b30, 0xc0019e4510, 0xc001898b20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2288 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4159) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b80c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b80c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898b80, 0xc0019e4510, 0xc001898b70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2289 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b81e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b81e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898bb0, 0xc0019e4510, 0xc001898ba0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2290 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4154) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b82a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b82a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898c40, 0xc0019e4510, 0xc001898c30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2291 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4162) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8300) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898c60, 0xc0019e4510, 0xc001898c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2292 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4160) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8330) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898c80, 0xc0019e4510, 0xc001898c70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2293 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8360) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ca0, 0xc0019e4510, 0xc001898c90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2294 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4156) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8390) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898cc0, 0xc0019e4510, 0xc001898cb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2295 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b83c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b83c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ce0, 0xc0019e4510, 0xc001898cd0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2296 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4143) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b83f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b83f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898d00, 0xc0019e4510, 0xc001898cf0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2297 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4141) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8420) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898d20, 0xc0019e4510, 0xc001898d10) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2298 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4151) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8450) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898d40, 0xc0019e4510, 0xc001898d30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2299 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4149) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898d60, 0xc0019e4510, 0xc001898d50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2300 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x41a6) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc001575560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc001575560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898dc0, 0xc0019e4510, 0xc001898db0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2301 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4166) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b84e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b84e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898e90, 0xc0019e4510, 0xc001898e50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2302 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8510) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898eb0, 0xc0019e4510, 0xc001898ea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2303 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b85d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b85d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ed0, 0xc0019e4510, 0xc001898ec0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2304 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x414f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b86c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b86c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ef0, 0xc0019e4510, 0xc001898ee0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2305 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4153) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b86f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b86f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898f10, 0xc0019e4510, 0xc001898f00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2306 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898f30, 0xc0019e4510, 0xc001898f20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2307 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898f70, 0xc0019e4510, 0xc001898f40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2308 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b87b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b87b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898f90, 0xc0019e4510, 0xc001898f80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2309 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4174) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b87e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b87e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898fb0, 0xc0019e4510, 0xc001898fa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2310 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4158) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8810) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898fd0, 0xc0019e4510, 0xc001898fc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2311 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4173) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8870) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001898ff0, 0xc0019e4510, 0xc001898fe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2312 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x415d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b88a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b88a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899010, 0xc0019e4510, 0xc001899000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2313 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4161) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b88d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b88d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899030, 0xc0019e4510, 0xc001899020) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2314 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4167) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899050, 0xc0019e4510, 0xc001899040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2315 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4169) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899070, 0xc0019e4510, 0xc001899060) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2316 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4188) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8990) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899090, 0xc0019e4510, 0xc001899080) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2317 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8ab0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018990b0, 0xc0019e4510, 0xc0018990a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2318 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8bd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8bd0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018990d0, 0xc0019e4510, 0xc0018990c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2319 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4176) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018990f0, 0xc0019e4510, 0xc0018990e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2320 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4180) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899110, 0xc0019e4510, 0xc001899100) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2321 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899140, 0xc0019e4510, 0xc001899130) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2322 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4171) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8cc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899170, 0xc0019e4510, 0xc001899150) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2323 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x416f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8d50) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018991a0, 0xc0019e4510, 0xc001899180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2324 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4164) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b84b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b84b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018991c0, 0xc0019e4510, 0xc0018991b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2325 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4170) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8de0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018991f0, 0xc0019e4510, 0xc0018991e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2326 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899240, 0xc0019e4510, 0xc001899230) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2327 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417b) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8ea0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899260, 0xc0019e4510, 0xc001899250) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2328 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x417a) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8ed0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899280, 0xc0019e4510, 0xc001899270) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2329 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4165) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8f00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018992a0, 0xc0019e4510, 0xc001899290) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2330 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4175) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8f30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018992c0, 0xc0019e4510, 0xc0018992b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2331 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4192) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b8ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b8ff0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018992e0, 0xc0019e4510, 0xc0018992d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2332 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4178) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b90e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b90e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899300, 0xc0019e4510, 0xc0018992f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2333 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4179) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b9110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b9110) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899320, 0xc0019e4510, 0xc001899310) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2334 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4187) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b9140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b9140) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899340, 0xc0019e4510, 0xc001899330) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2335 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4168) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b9170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b9170) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899360, 0xc0019e4510, 0xc001899350) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2336 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001742950, 0x4186) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001742940) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).worker(0xc001080180, 0x489e1a0, 0xc0008b91a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:192 +0x7c github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Start.func2(0x489e1a0, 0xc0008b91a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:161 +0x3e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899380, 0xc0019e4510, 0xc001899370) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2337 [semacquire]: sync.runtime_Semacquire(0xc0010801f8) /usr/local/go/src/runtime/sema.go:56 +0x39 sync.(*WaitGroup).Wait(0xc0010801f0) /usr/local/go/src/sync/waitgroup.go:130 +0x65 github.com/cockroachdb/cockroach/pkg/storage.(*raftScheduler).Wait(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/scheduler.go:167 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018993a0, 0xc0019e4510, 0xc001899390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2338 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).raftTickLoop(0xc00084c700, 0x489e1a0, 0xc0008b9230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3769 +0x264 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018993c0, 0xc0019e4510, 0xc0018993b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2339 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).coalescedHeartbeatsLoop(0xc00084c700, 0x489e1a0, 0xc0008b9260) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3825 +0x129 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018993e0, 0xc0019e4510, 0xc0018993d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2340 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).Start.func2(0x489e1a0, 0xc0008b9290) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1417 +0xe6 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899430, 0xc0019e4510, 0xc0002c92c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2341 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc0008b93b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899440, 0xc0019e4510, 0xc001a95220) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2342 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc0008b93e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899450, 0xc0019e4510, 0xc001a95270) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2343 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startGossip.func4(0x489e1a0, 0xc0008b95f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1558 +0x350 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899460, 0xc0019e4510, 0xc001a952c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2345 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startLeaseRenewer.func1(0x489e1a0, 0xc0008b9b00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1607 +0x2ea github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018994a0, 0xc0019e4510, 0xc001899490) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2346 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0018ae390, 0x9d) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0018ae380) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0005c8230, 0x489e1a0, 0xc000ac6930, 0xc002258240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func1(0x489e1a0, 0xc000ac6930) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1628 +0x57 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc000ac6930, 0xc00192a240, 0x25, 0x0, 0x0, 0xc0002c92e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2347 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*Store).startClosedTimestampRangefeedSubscriber.func2(0x489e1a0, 0xc0008b9b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:1636 +0x2db github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018994b0, 0xc0019e4510, 0xc0002c9300) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2348 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*StoreRebalancer).Start.func1(0x489e1a0, 0xc0008b9b90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_rebalancer.go:182 +0x181 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018994c0, 0xc0019e4510, 0xc0002c9320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2349 [select]: github.com/cockroachdb/cockroach/pkg/storage/compactor.(*Compactor).Start.func1.1(0x489e1a0, 0xc0008b9bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/compactor/compactor.go:123 +0x148 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018994d0, 0xc0019e4510, 0xc0002c9340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2350 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startComputePeriodicMetrics.func1(0x489e1a0, 0xc000930300) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:719 +0x11d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018995b0, 0xc0019e4510, 0xc0002c9580) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2351 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Node).startGossip.func1(0x489e1a0, 0xc0008b99e0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:684 +0x23e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899640, 0xc0019e4510, 0xc0002c9600) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2352 [select]: github.com/cockroachdb/cockroach/pkg/server.periodicallyPersistHLCUpperBound(0xc001845a90, 0xc001e1e2a0, 0xc0018996a0, 0x40e5438, 0xc000f743c0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1014 +0x18b github.com/cockroachdb/cockroach/pkg/server.(*Server).startPersistingHLCUpperBound.func2(0x489e1a0, 0xc00158d3e0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1088 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018996b0, 0xc0019e4510, 0xc000ac6e40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2353 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).refreshSettings.func2(0x489e1a0, 0xc00158d350) /go/src/github.com/cockroachdb/cockroach/pkg/server/settingsworker.go:110 +0x107 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018996c0, 0xc0019e4510, 0xc0002c9680) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2354 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSampleEnvironment.func1(0x489e1a0, 0xc0015827b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:2024 +0x1fc github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018997a0, 0xc0019e4510, 0xc000ac6f30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2355 [select]: github.com/cockroachdb/cockroach/pkg/ts.(*poller).start.func1(0x489e1a0, 0xc0009304b0) /go/src/github.com/cockroachdb/cockroach/pkg/ts/db.go:166 +0x112 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018997c0, 0xc0019e4510, 0xc0018997b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2356 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*SchemaChangeManager).Start.func1(0x489e1a0, 0xc00158d6e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/schema_changer.go:1991 +0x80e github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018997d0, 0xc0019e4510, 0xc0002c9760) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2357 [select]: github.com/cockroachdb/cockroach/pkg/sql/distsqlrun.(*flowScheduler).Start.func1(0x489e1a0, 0xc00158d6b0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/distsqlrun/flow_scheduler.go:147 +0x14f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899820, 0xc0019e4510, 0xc001899810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2358 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).Start.func1(0x489e1a0, 0xc001582750) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:324 +0xfa github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899830, 0xc0019e4510, 0xc0002c97c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2359 [select]: github.com/cockroachdb/cockroach/pkg/sql.(*Server).PeriodicallyClearSQLStats.func1(0x489e1a0, 0xc0015827e0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:682 +0x234 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001899850, 0xc0019e4510, 0xc0002c97e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2360 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*NodeLiveness).StartHeartbeat.func1(0x489e1a0, 0xc000ac71d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/node_liveness.go:488 +0x479 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0018999d0, 0xc0019e4510, 0xc001a956d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2366 [select]: github.com/cockroachdb/cockroach/pkg/server.(*Server).startSystemLogsGC.func1(0x489e1a0, 0xc001d6d200) /go/src/github.com/cockroachdb/cockroach/pkg/server/server_systemlog_gc.go:171 +0x536 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016ee320, 0xc0019e4510, 0xc001a6d180) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2605 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00340a640, 0xc003626410, 0x5, 0x5, 0x96, 0xc0020ef1d8, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00340a640, 0xc003626410, 0x5, 0x5, 0xc11b10, 0xc0035d0200, 0xc0039d3858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0018275f0, 0xc003626410, 0x5, 0x5, 0xc00694df60, 0xc0039d3880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0018275f0, 0xc003626410, 0x5, 0x5, 0x5, 0xc00694df60, 0x20, 0x18) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0011cb300, 0xc003626410, 0x5, 0x5, 0xc006340d80, 0x40, 0xc000544000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003626400, 0x7fffffff, 0xc0e586, 0xc000f91e50, 0x58, 0x58, 0xc007599ce0, 0xc0016eff70) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003626400, 0xc0011cb300, 0x0, 0x0, 0x7fffffff, 0xc0039d3b98, 0x4886820, 0x72b56a8, 0x40, 0xc001a1a09f, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003626400, 0x7f6d0c1990d0, 0xc000292db0, 0xc0011cb300, 0x0, 0x0, 0x3de9580, 0xc004df7050, 0x7fffffff, 0xc0039d3b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0019f7300, 0x3de9580, 0xc004df7050, 0xc0039d3b98, 0x0, 0xc004df7050) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0019f7300, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc001214900, 0xc0039d3ce0, 0xc0039d3cd0, 0xc00433bd80, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc001214900, 0x3de9580, 0xc004df7050, 0xc00694df20, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc000260c10, 0xc00694df20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc001827530) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260b80, 0xc000a30090, 0xc0036262a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2368 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc33d8, 0x72, 0x0) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a498, 0x72, 0x0, 0x0, 0x3f80125) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Accept(0xc000f7a480, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:384 +0x1ba net.(*netFD).accept(0xc000f7a480, 0xc001d66501, 0x0, 0x0) /usr/local/go/src/net/fd_unix.go:238 +0x42 net.(*TCPListener).accept(0xc000f94730, 0xc001412e68, 0xc001412e70, 0x38) /usr/local/go/src/net/tcpsock_posix.go:139 +0x32 net.(*TCPListener).Accept(0xc000f94730, 0x40e2110, 0xc0019e4510, 0x489e1a0, 0xc001d6c7b0) /usr/local/go/src/net/tcpsock.go:260 +0x48 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith(0xc001d665b8, 0x489e1a0, 0xc001d6c7b0, 0xc0019e4510, 0x4886160, 0xc000f94730, 0xc001d6c7e0, 0xc001d6c7b0, 0xc001b77f68) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:126 +0x50 github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1(0x489e1a0, 0xc001d6c7b0) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1809 +0xdd github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0016ee3a0, 0xc0019e4510, 0xc001a1f380) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 1116 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003360080, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0035824e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc003192000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2403 [select]: database/sql.(*DB).connectionOpener(0xc001a33d40, 0x489e0e0, 0xc001061840) /usr/local/go/src/database/sql/sql.go:1000 +0xe8 created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:670 +0x15e goroutine 2404 [select]: database/sql.(*DB).connectionResetter(0xc001a33d40, 0x489e0e0, 0xc001061840) /usr/local/go/src/database/sql/sql.go:1013 +0xfb created by database/sql.OpenDB /usr/local/go/src/database/sql/sql.go:671 +0x194 goroutine 2914 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0002a8960, 0x48fa580, 0xc001dbd7e0, 0x485c120, 0xc0002a8960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001d4ecc0, 0x0, 0xc002c85560) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001d4ecc0, 0xc003213320, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0002a8960, 0x48eb940, 0xc001d4ecc0, 0xc003213320, 0x40e1ba8, 0x489e1a0, 0xc00321ffb0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc002d8a180, 0xc001633600, 0xc000c58270, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc002d8a180, 0xc001633600, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0005e9b30, 0xc000c12a80, 0x48fef60, 0xc002d8a180, 0xc001633600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2406 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2d58, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001d93418, 0x72, 0x700, 0x719, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001d93400, 0xc00285f800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001d93400, 0xc00285f800, 0x719, 0x719, 0x203001, 0x48314e0, 0x72b4ea0) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001d665f8, 0xc00285f800, 0x719, 0x719, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc005edcac0, 0xc00285f800, 0x719, 0x719, 0x0, 0x48317a0, 0xc00469d618) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0019dc258, 0x482c340, 0xc005edcac0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0019dc000, 0x48317e0, 0xc001d665f8, 0x5, 0xc001d665f8, 0xc0055d1e00) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0019dc000, 0x40e5100, 0xc0019dc138, 0xed500c078) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0019dc000, 0xc002b93000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*readTimeoutConn).Read(0xc0003c39e0, 0xc002b93000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:1896 +0x110 bufio.(*Reader).fill(0xc00157d248) /usr/local/go/src/bufio/bufio.go:100 +0x10f bufio.(*Reader).ReadByte(0xc00157d248, 0x1, 0x1, 0x2) /usr/local/go/src/bufio/bufio.go:249 +0x39 github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase.(*ReadBuffer).ReadTypedMsg(0xc00157d9a0, 0x486e520, 0xc00157d248, 0x4860120, 0x72b56a8, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/pgwirebase/encoding.go:118 +0x35 github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).serveImpl(0xc00157d200, 0x489e0e0, 0xc001061f00, 0xc0016660a0, 0xc001960f00, 0x5400, 0x15000, 0xc001aab5d8, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:313 +0x474 github.com/cockroachdb/cockroach/pkg/sql/pgwire.serveConn(0x489e0e0, 0xc001061f00, 0x48f9e60, 0xc0019dc000, 0xc002d4c00d, 0x4, 0xc001607710, 0x485eca0, 0xc00116c300, 0x4000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:168 +0x18d github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*Server).ServeConn(0xc001aab440, 0x489e0e0, 0xc001061f00, 0x48f9e60, 0xc0019dc000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/pgwire/server.go:536 +0x87e github.com/cockroachdb/cockroach/pkg/server.(*Server).startServeSQL.func1.1(0x48faa00, 0xc001d665f8) /go/src/github.com/cockroachdb/cockroach/pkg/server/server.go:1813 +0x165 github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith.func1(0xc0019e4510, 0x489e1a0, 0xc001d6c7b0, 0xc001d665b8, 0x48faa00, 0xc001d665f8, 0xc001d6c7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:147 +0xa9 created by github.com/cockroachdb/cockroach/pkg/util/netutil.(*Server).ServeWith /go/src/github.com/cockroachdb/cockroach/pkg/util/netutil/net.go:144 +0x22b goroutine 10229 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0049218c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2834 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001ea0dc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2482 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2c88, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001022118, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001022100, 0xc0017f2800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001022100, 0xc0017f2800, 0x791, 0x791, 0x203000, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e14010, 0xc0017f2800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00367e600, 0xc0017f2800, 0x791, 0x791, 0x7c, 0x482c600, 0xc002f23a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000a2ecd8, 0x482c340, 0xc00367e600, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000a2ea80, 0x48317e0, 0xc002e14010, 0x5, 0xc002e14010, 0xc005b3f560) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000a2ea80, 0x40e5100, 0xc000a2ebb8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000a2ea80, 0xc0030fe000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002b91bc0, 0xc00174dc38, 0x9, 0x9, 0xc0064b3eb8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002b91bc0, 0xc00174dc38, 0x9, 0x9, 0x9, 0xc0009bc2d0, 0x485e320, 0xc002fd9cc0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00174dc38, 0x9, 0x9, 0x482c140, 0xc002b91bc0, 0x0, 0xc000000000, 0xc002d1fd00, 0x5b3f500) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00174dc00, 0xc005b3f530, 0xc005b3f530, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000b51180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2483 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc000f91e40, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002b91c80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000b51180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 2484 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001119e00, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002b91ce0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003110000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2485 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003110000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2486 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2bb8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a018, 0x72, 0x4900, 0x4978, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000f7a000, 0xc0031f0000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000f7a000, 0xc0031f0000, 0x4978, 0x4978, 0xc00140f938, 0x4901, 0xc000856f80) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e62000, 0xc0031f0000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002e64000, 0xc0031f0000, 0x4978, 0x4978, 0xc0031f000d, 0x5e, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0036ad280, 0xc0031f0000, 0x4978, 0x4978, 0x7b, 0x482c600, 0xc00140fa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000857058, 0x482c340, 0xc0036ad280, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000856e00, 0x7f6d0700c008, 0xc002e64000, 0x5, 0xc002e64000, 0x5e) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000856e00, 0x40e5100, 0xc000856f38, 0x7) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000856e00, 0xc003106000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002e64720, 0xc0003e11b8, 0x9, 0x9, 0xc0033ec288, 0xc005a9c060, 0x55) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002e64720, 0xc0003e11b8, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003e11b8, 0x9, 0x9, 0x482c140, 0xc002e64720, 0x0, 0x700000000, 0xc000b24e00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e1180, 0xc0015fcc00, 0xc0015fcc00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003110000, 0xc0016d4b10, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc003110000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc003110000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2739 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0014f0780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 1118 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc000527210, 0x9f) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc000527200) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0006964d0, 0x489e1a0, 0xc0031c0480, 0xc003582600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc0031c0480) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e1a0, 0xc0031c0480, 0xc0003f4300, 0x1d, 0x0, 0x0, 0xc00333e340) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2619 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002d9c7d0, 0xc0019e0cd0, 0x5, 0x5, 0x96, 0xc005da7bf8, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002d9c7d0, 0xc0019e0cd0, 0x5, 0x5, 0xc11b10, 0xc000390380, 0xc003a41858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00171f080, 0xc0019e0cd0, 0x5, 0x5, 0xc003f19920, 0xc003a41880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00171f080, 0xc0019e0cd0, 0x5, 0x5, 0x5, 0xc0011d4600, 0xc003a41948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f0100, 0xc0019e0cd0, 0x5, 0x5, 0x40, 0x40, 0xc00092e700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0019e0cc0, 0x7fffffff, 0xc0e5c1, 0xc002e24000, 0x58, 0x58, 0xc006347ce0, 0xc001dbc500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0019e0cc0, 0xc0010f0100, 0x0, 0x0, 0x7fffffff, 0xc003a41b98, 0x4886820, 0x72b56a8, 0x0, 0xc006fa7a6b, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0019e0cc0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f0100, 0x0, 0x0, 0x3de9580, 0xc006000f90, 0x7fffffff, 0xc003a41b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001568280, 0x3de9580, 0xc006000f90, 0xc003a41b98, 0x0, 0xc006000f90) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001568280, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d545a0, 0xc003a41ce0, 0xc003a41cd0, 0x72b56a8, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d545a0, 0x3de9580, 0xc006000f90, 0xc003f198e0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0019b40b0, 0xc003f198e0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc00171efc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffb30, 0xc000937c20, 0xc0019e0240) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2500 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc002d94640, 0xc00031a610, 0x5, 0x5, 0x0, 0x0, 0xc002f9b200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002d94640, 0xc00031a610, 0x5, 0x5, 0xc11b10, 0xc0033640a0, 0xc003357af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0017f0510, 0xc00031a610, 0x5, 0x5, 0x3a, 0xc003357b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0017f0510, 0xc00031a610, 0x5, 0x5, 0x5, 0xc003357bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0015c4500, 0xc00031a610, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00031a600, 0x7fffffff, 0x30, 0xc004ecdc80, 0x8b0000c0016d4690, 0x3, 0xc001d0b180, 0x7f6d0c2746d0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00031a600, 0xc0015c4500, 0x0, 0x0, 0x7fffffff, 0xc003357e30, 0x4886820, 0x72b56a8, 0x8, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00031a600, 0x7f6d0c1990d0, 0xc000292db0, 0xc0015c4500, 0x0, 0x0, 0x3de86e0, 0xc0054eb780, 0x7fffffff, 0xc003357e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc003366180, 0x3de86e0, 0xc0054eb780, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0002fa150, 0x489e1a0, 0xc0017f05d0, 0xc002f9b200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0002fa150, 0xc0002a8960, 0x489e1a0, 0xc0017f05d0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc0017f05d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0002fa170, 0xc000a30090, 0xc0017f05a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 237946 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc007828300, 0xc0060a9d09, 0xc00719fa90) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0067e4800, 0x3e99440, 0xc007eb3da0, 0xc00719fae8, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0067e4800, 0xc0068b02a0, 0x6d) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003f73320, 0xc00719fc30, 0xc00719fc20, 0xc004501020, 0xc0060a9d09) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003f73320, 0x3e99440, 0xc007eb3da0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e1a0, 0xc006456600, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0067e4780, 0x3e99440, 0xc007eb3da0, 0xc001ce2000, 0xc003a324c0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e1a0, 0xc006456600, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0067e4780, 0x3e99440, 0xc007eb3da0, 0xc001ce2000, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc001ce2000, 0x489e1a0, 0xc006456600, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0067e4780, 0x3e99440, 0xc007eb3da0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc00075c238, 0x489e1a0, 0xc006456600, 0xc0067e4780, 0x0, 0x0, 0x0, 0x2, 0x489e1a0, 0xc006456600) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006712380, 0x489e1a0, 0xc006456600, 0x2, 0x485a560, 0xc00075c238, 0x0, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006712380, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x200000002, 0x3, 0xc00601e85c, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc001c55560, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc001d92960, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc001d92900, 0x489e1a0, 0xc006456600, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e1a0, 0xc006456600, 0xc0071a1ba0, 0xc004856680, 0xc005b76700, 0xc00773fb58) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc001d92900, 0x489e1a0, 0xc006456600, 0xc004856680, 0xc00773fb58, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminChangeReplicas(0xc001d92900, 0x489e1a0, 0xc006456600, 0x3d0b580, 0xc00652e160, 0xc1, 0xc00773fb58, 0x3, 0x8, 0xc00773fb60, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:582 +0x268 github.com/cockroachdb/cockroach/pkg/storage.(*Store).AdminRelocateRange(0xc00084c700, 0x489e1a0, 0xc006456600, 0xc1, 0xc00773fb58, 0x3, 0x8, 0xc00773fb60, 0x3, 0x8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:2027 +0xbc3 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc003c36800, 0x489e1a0, 0xc006456600, 0xc0073da680, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1335 +0x803 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc003c36800, 0x489e1a0, 0xc0064565d0, 0xc1, 0x15c106cd484c2408, 0x0, 0x300000003, 0x2, 0xc0003198ec, 0xc1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc003c36800, 0x489e1a0, 0xc0064565d0, 0x15c106cd484c2408, 0x0, 0x300000003, 0x2, 0xc0003198ec, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc00084c700, 0x489e1a0, 0xc006456570, 0x15c106cd484c2408, 0x0, 0x300000003, 0x2, 0xc0003198ec, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc00177b5e0, 0x489e1a0, 0xc006456570, 0x0, 0x0, 0x300000003, 0x2, 0xc0003198ec, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc006456540, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc0019e4510, 0x489e1a0, 0xc006456540, 0x3f956b7, 0x10, 0xc007a1d710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc001960000, 0x489e1a0, 0xc006456540, 0xc0073da600, 0xc006456540, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc001960000, 0x489e1a0, 0xc006456510, 0xc0073da600, 0xc001960000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc006456510, 0x3f4fce0, 0xc0073da600, 0x3f4fce0, 0xc0073da600, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc006456510, 0x3f4fce0, 0xc0073da600, 0xc0073f5fe0, 0xc007a1e000, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc006456510, 0x3f4fce0, 0xc0073da600, 0xc0073f5fe0, 0xc007a1e000, 0x6ff238, 0x20, 0x3b5a920, 0xc005227201) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc006456510, 0x3f4fce0, 0xc0073da600, 0xc0073f5fe0, 0xc007a1e000, 0xc004d28bc8, 0x7edeba, 0x3d2f600, 0xc006456510) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc001960000, 0x489e1a0, 0xc006456510, 0xc005227200, 0xc000261b90, 0x489e1a0, 0xc006456510, 0xc0064b8c00, 0x25) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc006041c00, 0xc000c3c0c0, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc006041c00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f2580, 0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc006041c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 136987 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc000969f80) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e1a0, 0xc000969f80, 0xc00574ca20, 0x2a, 0x0, 0x0, 0xc0017b48a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2795 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001c14740, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003876ba0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc0022d5200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2885 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0016a99c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 236399 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc4, 0x490bae0, 0xc005082a80, 0x0, 0x1, 0xc001854e18, 0x2b9744f) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc4, 0xc005570fc0, 0xc0008af800, 0x15c106cd37020ac3) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0047ca400, 0x33, 0x0, 0x0, 0xc002d5d140) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 236002 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc007095aa0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc1, 0xc0051c5d40, 0xc00285c000, 0x15c106cd30a8673a) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003e16880, 0x33, 0x0, 0x0, 0xc0064b79e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2670 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003198880, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002e20360, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00293e540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 238982 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00174c380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc0070ee780, 0x0, 0x1, 0xc000cae6c0, 0xbab23ccf) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc00215c8f0, 0x489e120, 0xc00010c010, 0xc1, 0xc00785b8c0, 0xc0035dc800, 0x15c106cd4ca161af) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e120, 0xc00010c010, 0xc00776cdc0, 0x33, 0x0, 0x0, 0xc0074dd440) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 5805 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce598, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc002982498, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc002982480, 0xc001e99000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc002982480, 0xc001e99000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e62110, 0xc001e99000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc005640a40, 0xc001e99000, 0x791, 0x791, 0x101, 0x482c600, 0xc004362a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000b533d8, 0x482c340, 0xc005640a40, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000b53180, 0x48317e0, 0xc002e62110, 0x5, 0xc002e62110, 0xc003673d10) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000b53180, 0x40e5100, 0xc000b532b8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000b53180, 0xc003022000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00459c060, 0xc0003e1618, 0x9, 0x9, 0xc006244298, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00459c060, 0xc0003e1618, 0x9, 0x9, 0x9, 0xc004e60990, 0x485e320, 0xc006a48710) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc0003e1618, 0x9, 0x9, 0x482c140, 0xc00459c060, 0x0, 0xc000000000, 0xc004d02300, 0x3673c00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc0003e15e0, 0xc003673ce0, 0xc003673ce0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001084540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2478 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc001708840) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2122 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc001915e00, 0x7f6d05d41348, 0xc0016b68a0, 0x7f6d05d41348, 0xc0016b68a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc001915e00, 0x48fa760, 0xc0016b68a0, 0x8a, 0x8b) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc00341a0c0, 0x0, 0xc001894840) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc00341a0c0, 0xc0003cd320, 0x40e1e60, 0x3c89ac0, 0x47cf94f) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003fcf60, 0x48eb940, 0xc00341a0c0, 0xc0003cd320, 0x40e1e60, 0x489e1a0, 0xc0015ef950) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc001892180, 0xc000b25200, 0xc00111bb30, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc001892180, 0xc000b25200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000318e10, 0xc00193c600, 0x48fef60, 0xc001892180, 0xc000b25200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2796 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc0022d5200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2606 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0031ef360, 0xc003628490, 0x5, 0x5, 0x94, 0xc005da6208, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0031ef360, 0xc003628490, 0x5, 0x5, 0xc11b10, 0xc001d73c80, 0xc001b7d858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001a36bd0, 0xc003628490, 0x5, 0x5, 0xc00505fe60, 0xc001b7d880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001a36bd0, 0xc003628490, 0x5, 0x5, 0x5, 0xc0009a5b00, 0xc001b7d948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00244d000, 0xc003628490, 0x5, 0x5, 0xc00723af80, 0x40, 0xc001ca2000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003628480, 0x7fffffff, 0xc0e5c1, 0xc000f75c20, 0x58, 0x58, 0xc002fcafc0, 0xc000278200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003628480, 0xc00244d000, 0x0, 0x0, 0x7fffffff, 0xc001b7db98, 0x4886820, 0x72b56a8, 0x40, 0xc006f2faf5, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003628480, 0x7f6d0c1990d0, 0xc000292db0, 0xc00244d000, 0x0, 0x0, 0x3de9580, 0xc006dbf620, 0x7fffffff, 0xc001b7db98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc000f7bc00, 0x3de9580, 0xc006dbf620, 0xc001b7db98, 0x0, 0xc006dbf620) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc000f7bc00, 0x5, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc003324240, 0xc001b7dce0, 0xc001b7dcd0, 0xc005df2cc0, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc003324240, 0x3de9580, 0xc006dbf620, 0xc00505fe20, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0015a98b0, 0xc00505fe20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc001a36b10) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260bc0, 0xc000a30090, 0xc0036262c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 3104 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001b18700, 0xc00324b200, 0x489e1a0, 0xc0032c1b90) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2101 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc000383de0, 0x7f6d05d41348, 0xc0003103c0, 0x7f6d05d41348, 0xc0003103c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc000383de0, 0x48fa760, 0xc0003103c0, 0xc02f00010301, 0xc0003103c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc0009ce0c0, 0x0, 0xc001bd4b40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc0009ce0c0, 0xc000247a40, 0x40e1e60, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0002bc380, 0x48eb940, 0xc0009ce0c0, 0xc000247a40, 0x40e1e60, 0x489e1a0, 0xc0015fc390) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc000f84000, 0xc000c30750, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc000f84000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001d07520, 0xc000c12a80, 0x48fef60, 0xc0009ed080, 0xc000f84000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2618 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002d9c0f0, 0xc0019e04d0, 0x5, 0x5, 0x95, 0xc006053a60, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002d9c0f0, 0xc0019e04d0, 0x5, 0x5, 0xc11b10, 0xc000390120, 0xc001bc5858) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00171e660, 0xc0019e04d0, 0x5, 0x5, 0xc00682a060, 0xc001bc5880, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00171e660, 0xc0019e04d0, 0x5, 0x5, 0x5, 0xc0015cb800, 0xc001bc5948, 0x6f819c) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0008bda00, 0xc0019e04d0, 0x5, 0x5, 0xc0032cfe60, 0x40, 0xc001a65c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0019e04c0, 0x7fffffff, 0xc0e5c1, 0xc000a9ccc0, 0x58, 0x58, 0xc002999f80, 0xc0019b4880) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0019e04c0, 0xc0008bda00, 0x0, 0x0, 0x7fffffff, 0xc001bc5b98, 0x4886820, 0x72b56a8, 0x40, 0xc0053e37a5, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0019e04c0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0008bda00, 0x0, 0x0, 0x3de9580, 0xc003c7efc0, 0x7fffffff, 0xc001bc5b98, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001568000, 0x3de9580, 0xc003c7efc0, 0xc001bc5b98, 0x0, 0xc003c7efc0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001568000, 0x5, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d54000, 0xc001bc5ce0, 0xc001bc5cd0, 0xc00415f280, 0x20) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d54000, 0x3de9580, 0xc003c7efc0, 0xc00682a020, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb.(*closedTimestampGetClient).Recv(0xc0010ffb90, 0xc00682a020, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:179 +0x62 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Clients).getOrCreateClient.func1(0x489e1a0, 0xc00171e540) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/clients.go:142 +0x3eb github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffa90, 0xc000937c20, 0xc0019e0040) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2622 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc00172aa80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2607 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0009f2e00, 0xc001214900, 0x489e1a0, 0xc0018274d0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2608 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc001827680) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000260c20, 0xc000a30090, 0xc003626420) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6019 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc0041ba000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 236820 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6e, 0x490bae0, 0xc0026afaa0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6e, 0xc0055c05a0, 0xc001858f38, 0xc00685f0a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003edec40, 0x33, 0x0, 0x0, 0xc004d4e840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 2623 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc002e2c180, 0xc0002342d0, 0xf, 0xc002974240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc00171e900) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0010ffdd0, 0xc000937c20, 0xc002d9c230) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 5685 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc003cb3200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2740 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2a18, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001568218, 0x72, 0x6300, 0x6333, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001568200, 0xc004236000, 0x6333, 0x6333, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001568200, 0xc004236000, 0x6333, 0x6333, 0xc001a4e938, 0x6301, 0xc002256880) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc002e620f0, 0xc004236000, 0x6333, 0x6333, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002e2c660, 0xc004236000, 0x6333, 0x6333, 0xc00423600d, 0x5c, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0036ade20, 0xc004236000, 0x6333, 0x6333, 0x79, 0x482c600, 0xc001a4ea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc002256958, 0x482c340, 0xc0036ade20, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc002256700, 0x7f6d0700c008, 0xc002e2c660, 0x5, 0xc002e2c660, 0x5c) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc002256700, 0x40e5100, 0xc002256838, 0x7) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc002256700, 0xc001236000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001b06840, 0xc00174c2d8, 0x9, 0x9, 0xc002f2eb08, 0xc005a9c1e0, 0x53) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001b06840, 0xc00174c2d8, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00174c2d8, 0x9, 0x9, 0x482c140, 0xc001b06840, 0x0, 0x700000000, 0xc002d1e400, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00174c2a0, 0xc00079fbc0, 0xc00079fbc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc0014f0780, 0xc000a01bc0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc0014f0780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc0014f0780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2626 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002256000, 0xc000d545a0, 0x489e1a0, 0xc00171ef60) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2755 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc00171f1d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b40f0, 0xc000937c20, 0xc0019e0ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2756 [select]: github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get(0xc00039e140, 0x7f6d05d41348, 0xc0019b4350, 0x7f6d05d41348, 0xc0019b4350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:82 +0x2e0 github.com/cockroachdb/cockroach/pkg/storage/closedts/container.delayedServer.Get(0x1, 0x482f580, 0xc00039e140, 0x48fa760, 0xc0019b4350, 0xc02f00010301, 0xc0019b4350) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/container/container.go:96 +0x119 github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb._ClosedTimestamp_Get_Handler(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00092c000, 0x0, 0xc001b067e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/ctpb/entry.pb.go:195 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00092c000, 0xc0019e0fa0, 0x40e1e60, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3acf0c0, 0xc0003fc760, 0x48eb940, 0xc00092c000, 0xc0019e0fa0, 0x40e1e60, 0x489e1a0, 0xc00171f440) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc0010f0200, 0xc000c3c270, 0x6f9bf60, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc0010f0200, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f2580, 0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc0010f0200) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2757 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc0018ae390, 0x9c) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc0018ae380) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0005c8230, 0x489e1a0, 0xc00171f440, 0xc002e2cae0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc00171f440) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc00171f440, 0xc000390420, 0x1d, 0x0, 0x0, 0xc0019e0fe0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10233 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce258, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0035ff498, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0035ff480, 0xc004fcc800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0035ff480, 0xc004fcc800, 0x791, 0x791, 0x203002, 0x0, 0xc0034e99a8) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc001668758, 0xc004fcc800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc00844a400, 0xc004fcc800, 0x791, 0x791, 0x42, 0x482c600, 0xc0034e9a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003ecccd8, 0x482c340, 0xc00844a400, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003ecca80, 0x48317e0, 0xc001668758, 0x5, 0xc001668758, 0xc0083c3350) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003ecca80, 0x40e5100, 0xc003eccbb8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003ecca80, 0xc0048f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002a595c0, 0xc004944e38, 0x9, 0x9, 0xc00367d7f8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002a595c0, 0xc004944e38, 0x9, 0x9, 0x9, 0xc00597fec0, 0x485e320, 0xc0060a8050) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc004944e38, 0x9, 0x9, 0x482c140, 0xc002a595c0, 0x0, 0xc000000000, 0xc006c3e900, 0x83c3300) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc004944e00, 0xc0083c3320, 0xc0083c3320, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00492e700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 3105 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0036b19f0, 0xc00279cc30, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0036b19f0, 0xc00279cc30, 0x5, 0x5, 0xc11b10, 0xc001d72c00, 0xc00453f978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0032c1cb0, 0xc00279cc30, 0x5, 0x5, 0x74cff0, 0xc00453f9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0032c1cb0, 0xc00279cc30, 0x5, 0x5, 0x5, 0xf, 0xc00107e1e0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002700e00, 0xc00279cc30, 0x5, 0x5, 0x83e500, 0xc0062511f0, 0xc00453fa78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00279cc20, 0x7fffffff, 0x3c06120, 0xc00107e1e0, 0xc00453fa0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00279cc20, 0xc002700e00, 0x0, 0x0, 0x7fffffff, 0xc00453fcb8, 0x4886820, 0x72b56a8, 0x0, 0xc0021adc20, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00279cc20, 0x7f6d0c1990d0, 0xc000292db0, 0xc002700e00, 0x0, 0x0, 0x3de8800, 0xc002f2f340, 0x7fffffff, 0xc00453fcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0027b0080, 0x3de8800, 0xc002f2f340, 0xc00453fcb8, 0xc0059bc500, 0xc002f2f340) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0027b0080, 0xc000092f70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00324b200, 0xc00453fe00, 0xc00453fdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00324b200, 0x3de8800, 0xc002f2f340, 0x2c5e8e0, 0xc000932000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc001842fb0, 0x489e1a0, 0xc0032c1d40, 0xc004d3a400) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc001842fb0, 0xc002a4c240, 0xc001460780, 0x489e1a0, 0xc0032c1d40, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc0032c1d40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001842fd0, 0xc000937c20, 0xc0032c1ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2891 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc001460780, 0x48fa580, 0xc0019b5310, 0x485c120, 0xc001460780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00092c900, 0x0, 0xc002e20120) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00092c900, 0xc0002bca40, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00092c900, 0xc0002bca40, 0x40e1ba8, 0x489e1a0, 0xc001b2b710) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc002637380, 0xc0010f0b00, 0xc00111b2c0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc002637380, 0xc0010f0b00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001d711f0, 0xc00193c600, 0x48fef60, 0xc002637380, 0xc0010f0b00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 2892 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0034a54a0, 0xc0002bca30, 0x5, 0x5, 0x0, 0x0, 0xc005b30c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0034a54a0, 0xc0002bca30, 0x5, 0x5, 0xc11b10, 0xc0003917a0, 0xc002d37af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001b2b6b0, 0xc0002bca30, 0x5, 0x5, 0x39, 0xc002d37b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001b2b6b0, 0xc0002bca30, 0x5, 0x5, 0x5, 0xc002d37bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f0b00, 0xc0002bca30, 0x5, 0x5, 0x203000, 0x203000, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002bca20, 0x7fffffff, 0x732a2f, 0xc0009a5380, 0x4, 0xc002d37c70, 0x733ecb, 0xc006210d80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002bca20, 0xc0010f0b00, 0x0, 0x0, 0x7fffffff, 0xc002d37e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002bca20, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f0b00, 0x0, 0x0, 0x3de86e0, 0xc00030f1a0, 0x7fffffff, 0xc002d37e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00092c900, 0x3de86e0, 0xc00030f1a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0019b5310, 0x489e1a0, 0xc001b2b770, 0xc005b30c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0019b5310, 0xc001460780, 0x489e1a0, 0xc001b2b770, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc001b2b770) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5330, 0xc000937c20, 0xc001b2b740) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 9965 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc004980000, 0xc004020b40, 0x489e1a0, 0xc001799dd0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 1114 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc003192000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2889 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc002152580) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 2888 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0034a50e0, 0xc00038bcb0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0034a50e0, 0xc00038bcb0, 0x5, 0x5, 0xc11b10, 0xc000391680, 0xc003991978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001b2b3e0, 0xc00038bcb0, 0x5, 0x5, 0x74cff0, 0xc0039919a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001b2b3e0, 0xc00038bcb0, 0x5, 0x5, 0x5, 0xf, 0xc000079c80, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f0a00, 0xc00038bcb0, 0x5, 0x5, 0x83e500, 0xc005c6b7d0, 0xc003991a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00038bca0, 0x7fffffff, 0x3c06120, 0xc000079c80, 0xc003991a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00038bca0, 0xc0010f0a00, 0x0, 0x0, 0x7fffffff, 0xc003991cb8, 0x4886820, 0x72b56a8, 0x0, 0xc0063fda00, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00038bca0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f0a00, 0x0, 0x0, 0x3de8800, 0xc005cc3700, 0x7fffffff, 0xc003991cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001568c80, 0x3de8800, 0xc005cc3700, 0xc003991cb8, 0xc0038cc6e0, 0xc005cc3700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001568c80, 0xc000084f70, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d55440, 0xc003991e00, 0xc003991df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d55440, 0x3de8800, 0xc005cc3700, 0x2c5e8e0, 0xc000a15500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0019b51c0, 0x489e1a0, 0xc001b2b470, 0xc0033c9e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0019b51c0, 0xc001db86c0, 0xc0002a8960, 0x489e1a0, 0xc001b2b470, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc001b2b470) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b51e0, 0xc000a30090, 0xc001b2b410) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2887 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001b19c00, 0xc000d55440, 0x489e1a0, 0xc001b2b2c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2669 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2948, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0035e2118, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0035e2100, 0xc001bff000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0035e2100, 0xc001bff000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f948a8, 0xc001bff000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc004caafc0, 0xc001bff000, 0x791, 0x791, 0x75, 0x482c600, 0xc002136a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000cb1e58, 0x482c340, 0xc004caafc0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000cb1c00, 0x48317e0, 0xc000f948a8, 0x5, 0xc000f948a8, 0xc003d52de0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000cb1c00, 0x40e5100, 0xc000cb1d38, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000cb1c00, 0xc0022f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002e202a0, 0xc00111ded8, 0x9, 0x9, 0xc003399d38, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002e202a0, 0xc00111ded8, 0x9, 0x9, 0x9, 0xc003d52480, 0x485e320, 0xc00027a4b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111ded8, 0x9, 0x9, 0x482c140, 0xc002e202a0, 0x0, 0xc000000000, 0xc007c8c400, 0x3d52d00) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111dea0, 0xc003d52d20, 0xc003d52d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc00293e540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 2898 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0002a8960, 0x3, 0xc002e65e00, 0xc000223e00, 0x48fa520, 0xc0019b5650, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc0031964e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b55d0, 0xc000a30090, 0xc0031964b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2668 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc00293e540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 2967 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc003197a40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5ab0, 0xc0019e4510, 0xc003337000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2920 [IO wait]: internal/poll.runtime_pollWait(0x7f6d05dc2878, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0035e2198, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0035e2180, 0xc001bff800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0035e2180, 0xc001bff800, 0x75b, 0x75b, 0xc00140a938, 0x701, 0xc0015a6180) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f948b8, 0xc001bff800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc003145020, 0xc001bff800, 0x75b, 0x75b, 0xc001bff80d, 0x59, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc00030f0a0, 0xc001bff800, 0x75b, 0x75b, 0x76, 0x482c600, 0xc00140aa50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0015a6258, 0x482c340, 0xc00030f0a0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0015a6000, 0x7f6d0700c008, 0xc003145020, 0x5, 0xc003145020, 0x59) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0015a6000, 0x40e5100, 0xc0015a6138, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0015a6000, 0xc0022d8000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002e20180, 0xc00111dc38, 0x9, 0x9, 0xc0016a9c48, 0xc00336d8b0, 0x50) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002e20180, 0xc00111dc38, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00111dc38, 0x9, 0x9, 0x482c140, 0xc002e20180, 0x0, 0x300000000, 0xc0010f0b00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00111dc00, 0xc001b14d20, 0xc001b14d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc002637380, 0xc002d5def0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc002637380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc002637380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 2919 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc002637380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 2918 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003198840, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003351d40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc002637380) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 2963 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001e18e00, 0xc000d55b00, 0x489e1a0, 0xc0031965a0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 2964 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00340aaa0, 0xc0002bdbb0, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00340aaa0, 0xc0002bdbb0, 0x5, 0x5, 0xc11b10, 0xc000391d00, 0xc004c03978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0031966c0, 0xc0002bdbb0, 0x5, 0x5, 0x74cff0, 0xc004c039a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0031966c0, 0xc0002bdbb0, 0x5, 0x5, 0x5, 0xf, 0xc000079c80, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f1000, 0xc0002bdbb0, 0x5, 0x5, 0x83e500, 0xc0048dd6c0, 0xc004c03a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0002bdba0, 0x7fffffff, 0x3c06120, 0xc000079c80, 0xc004c03a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0002bdba0, 0xc0010f1000, 0x0, 0x0, 0x7fffffff, 0xc004c03cb8, 0x4886820, 0x72b56a8, 0x0, 0xc00518e560, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0002bdba0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f1000, 0x0, 0x0, 0x3de8800, 0xc00444c5c0, 0x7fffffff, 0xc004c03cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc001569200, 0x3de8800, 0xc00444c5c0, 0xc004c03cb8, 0xc002b28550, 0xc00444c5c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001569200, 0xc000092f70, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc000d55b00, 0xc004c03e00, 0xc004c03df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc000d55b00, 0x3de8800, 0xc00444c5c0, 0x2c5e8e0, 0xc000a15500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc0019b5650, 0x489e1a0, 0xc003196750, 0xc004c98dc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc0019b5650, 0xc000223e00, 0xc0002a8960, 0x489e1a0, 0xc003196750, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003196750) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0019b5670, 0xc000a30090, 0xc0031966f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2991 [chan receive]: github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Notify.func1(0x489e1a0, 0xc003505380) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:217 +0x147 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8b00, 0xc0019e4510, 0xc001a13ea0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2992 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc003a325c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 2993 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc0030d8660, 0xc000448e90, 0xf, 0xc00107fa40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc003505560) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8d10, 0xc0019e4510, 0xc001b38320) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 2994 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc0017098c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 3040 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc003640540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 3091 [sync.Cond.Wait]: runtime.goparkunlock(...) /usr/local/go/src/runtime/proc.go:307 sync.runtime_notifyListWait(0xc001bdce50, 0x9e) /usr/local/go/src/runtime/sema.go:510 +0xf9 sync.(*Cond).Wait(0xc001bdce40) /usr/local/go/src/sync/cond.go:56 +0x9e github.com/cockroachdb/cockroach/pkg/storage/closedts/provider.(*Provider).Subscribe(0xc0007844d0, 0x489e1a0, 0xc00333dc80, 0xc002639bc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/provider/provider.go:290 +0x436 github.com/cockroachdb/cockroach/pkg/storage/closedts/transport.(*Server).Get.func1(0x489e1a0, 0xc00333dc80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/closedts/transport/server.go:65 +0x50 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e1a0, 0xc00333dc80, 0xc0003d16a0, 0x1d, 0x0, 0x0, 0xc003074a20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 3061 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce808, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001c6b498, 0x72, 0x4900, 0x4978, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001c6b480, 0xc002684000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001c6b480, 0xc002684000, 0x4978, 0x4978, 0xc003805938, 0x4901, 0xc001ce3300) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0012507f8, 0xc002684000, 0x4978, 0x4978, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0030d8ea0, 0xc002684000, 0x4978, 0x4978, 0xc00268400d, 0x5b, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc00380a360, 0xc002684000, 0x4978, 0x4978, 0x78, 0x482c600, 0xc003805a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc001ce33d8, 0x482c340, 0xc00380a360, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc001ce3180, 0x7f6d0700c008, 0xc0030d8ea0, 0x5, 0xc0030d8ea0, 0x5b) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc001ce3180, 0x40e5100, 0xc001ce32b8, 0x5) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc001ce3180, 0xc00302a000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002612720, 0xc002ffc118, 0x9, 0x9, 0xc003d13d88, 0xc0032d4de0, 0x52) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002612720, 0xc002ffc118, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc002ffc118, 0x9, 0x9, 0x482c140, 0xc002612720, 0x0, 0x500000000, 0xc0037f7a00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc002ffc0e0, 0xc0025ed140, 0xc0025ed140, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00260a480, 0xc0025b1bf0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc00260a480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc00260a480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 3060 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc00260a480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 3059 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001743580, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0024b8ae0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc00260a480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 235327 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc2, 0x490bae0, 0xc007d608a0, 0x0, 0x1, 0x8, 0xc00571c130) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc2, 0xc000c42900, 0xc003c56800, 0x15c106cd2afd2b72) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00422f100, 0x33, 0x0, 0x0, 0xc00676a1b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 9778 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc002e1f220, 0xc002f63910, 0x5, 0x5, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002e1f220, 0xc002f63910, 0x5, 0x5, 0xc11b10, 0xc0003f5220, 0xc0034cbaf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003505a10, 0xc002f63910, 0x5, 0x5, 0x0, 0xc0034cbb20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003505a10, 0xc002f63910, 0x5, 0x5, 0x5, 0x0, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0017d2100, 0xc002f63910, 0x5, 0x5, 0x0, 0x300000003, 0x3) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc002f63900, 0x7fffffff, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc002f63900, 0xc0017d2100, 0x0, 0x0, 0x7fffffff, 0xc0034cbe30, 0x4886820, 0x72b56a8, 0x8, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc002f63900, 0x7f6d0c1990d0, 0xc000292db0, 0xc0017d2100, 0x0, 0x0, 0x3de86e0, 0xc0036add20, 0x7fffffff, 0xc0034cbe30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc0048b7200, 0x3de86e0, 0xc0036add20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc0015a8cd0, 0x489e1a0, 0xc003505ad0, 0xc003966a00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc0015a8cd0, 0xc0018a82d0, 0x489e1a0, 0xc003505ad0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003505ad0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0015a8d00, 0xc0019e4510, 0xc003505aa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 9777 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0018a82d0, 0x48fa580, 0xc0015a8cd0, 0x485c120, 0xc0018a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc0048b7200, 0x0, 0xc003851500) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc0048b7200, 0xc002f63920, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc0048b7200, 0xc002f63920, 0x40e1ba8, 0x489e1a0, 0xc003505a70) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc003e00600, 0xc0017d2100, 0xc0009a98f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc003e00600, 0xc0017d2100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001dc94f0, 0xc00262b200, 0x48fef60, 0xc003e00600, 0xc0017d2100) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 236116 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc007599440, 0x0, 0x1, 0x1, 0xc0008d0008) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc1, 0xc003960480, 0xc005fdf000, 0x15c106cd31433c4f) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003f6d240, 0x33, 0x0, 0x0, 0xc001d8a390) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6321 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc002256000, 0xc00073e5a0, 0x489e1a0, 0xc003af0f00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 10237 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce188, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0025abc18, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0025abc00, 0xc003593800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0025abc00, 0xc003593800, 0x75b, 0x75b, 0xc001d14938, 0x701, 0xc004078180) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110b48, 0xc003593800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc002f6a9c0, 0xc003593800, 0x75b, 0x75b, 0xc00359380d, 0x29, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc007124e20, 0xc003593800, 0x75b, 0x75b, 0x46, 0x482c600, 0xc001d14a50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc004078258, 0x482c340, 0xc007124e20, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc004078000, 0x7f6d0700c008, 0xc002f6a9c0, 0x5, 0xc002f6a9c0, 0x29) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc004078000, 0x40e5100, 0xc004078138, 0xc007693020) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc004078000, 0xc0048e0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc002a594a0, 0xc004944d58, 0x9, 0x9, 0xc003482108, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc002a594a0, 0xc004944d58, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc004944d58, 0x9, 0x9, 0x482c140, 0xc002a594a0, 0x0, 0x6500000000, 0xc0084fa100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc004944d20, 0xc003f5f0b0, 0xc003f5f0b0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc00368ad80, 0xc003f5f500, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc00368ad80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc00368ad80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 234655 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6a, 0x490bae0, 0xc0037e9c20, 0x0, 0x1, 0xc001b745e0, 0x7ee93c) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6a, 0xc0034deea0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0034186c0, 0x33, 0x0, 0x0, 0xc0045ea7e0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 240095 [runnable]: github.com/cockroachdb/cockroach/pkg/storage/engine._Cfunc_MVCCGet(0x7f6d02610440, 0xc003acb4f0, 0x5, 0x15c106cd58ea2581, 0x0, 0xc005292b40, 0x10, 0x0, 0x15c106cd58ea2581, 0x0, ...) _cgo_gotypes.go:1332 +0x6a github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCGet.func1(0xc005c20b48, 0xc003acb4f0, 0x5, 0x8, 0x15c106cd58ea2581, 0x0, 0x0, 0xc005292b40, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2382 +0x1a4 github.com/cockroachdb/cockroach/pkg/storage/engine.(*rocksDBIterator).MVCCGet(0xc005c20b48, 0xc003acb4f0, 0x5, 0x8, 0x15c106cd58ea2581, 0x0, 0x0, 0xc005292b40, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/rocksdb.go:2381 +0x10f github.com/cockroachdb/cockroach/pkg/storage/engine.MVCCGet(0x489e1a0, 0xc006eb3e00, 0x48eaaa0, 0xc005c20b40, 0xc003acb4f0, 0x5, 0x8, 0x15c106cd58ea2581, 0xc000000000, 0x2000000, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/engine/mvcc.go:726 +0x22b github.com/cockroachdb/cockroach/pkg/storage/batcheval.Get(0x489e1a0, 0xc006eb3e00, 0x4920740, 0xc005c20b40, 0x4933780, 0xc0008af000, 0x15c106cd58ea2581, 0x0, 0x100000001, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/batcheval/cmd_get.go:34 +0x19e github.com/cockroachdb/cockroach/pkg/storage.evaluateCommand(0x489e1a0, 0xc006eb3e00, 0x0, 0x0, 0x0, 0x4920740, 0xc005c20b40, 0x4933780, 0xc0008af000, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_evaluate.go:416 +0x235 github.com/cockroachdb/cockroach/pkg/storage.evaluateBatch(0x489e1a0, 0xc006eb3e00, 0x0, 0x0, 0x4920740, 0xc005c20b40, 0x4933780, 0xc0008af000, 0x0, 0xc006726d80, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_evaluate.go:262 +0x335 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeReadOnlyBatch(0xc0008af000, 0x489e1a0, 0xc006eb3e00, 0xc006726d80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_read.go:94 +0x552 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc0008af000, 0x489e1a0, 0xc006eb3da0, 0x6, 0x15c106cd58ea2581, 0x0, 0x100000001, 0x1, 0x0, 0x6, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:578 +0x2c9 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc0008af000, 0x489e1a0, 0xc006eb3da0, 0x15c106cd58ea2581, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a15500, 0x489e1a0, 0xc006eb3d10, 0x15c106cd58ea2581, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc0003b5f40, 0x489e1a0, 0xc006eb3d10, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc006eb3d10, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000a30090, 0x489e1a0, 0xc006eb3d10, 0x3f956b7, 0x10, 0xc0029ae478, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000786500, 0x489e1a0, 0xc006eb3d10, 0xc006726d00, 0xc006eb3d10, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000786500, 0x489e1a0, 0xc006eb3ce0, 0xc006726d00, 0xc006eb3c20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:540 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc007629f00, 0x489e1a0, 0xc006eb3ce0, 0x1, 0x485fe20, 0xc0003623a0, 0x0, 0x0, 0x100000001, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc007629f00, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x100000001, 0x1, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x6, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000c42000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/kv.(*txnLockGatekeeper).SendLocked(0xc005c14418, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:226 +0xe8 github.com/cockroachdb/cockroach/pkg/kv.(*txnMetricRecorder).SendLocked(0xc005c143e0, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_metric_recorder.go:48 +0xa6 github.com/cockroachdb/cockroach/pkg/kv.(*txnCommitter).SendLocked(0xc005c143b0, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_committer.go:126 +0x775 github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).sendLockedWithRefreshAttempts(0xc005c14330, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:198 +0x86 github.com/cockroachdb/cockroach/pkg/kv.(*txnSpanRefresher).SendLocked(0xc005c14330, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_span_refresher.go:154 +0xf1 github.com/cockroachdb/cockroach/pkg/kv.(*txnPipeliner).SendLocked(0xc005c14280, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_pipeliner.go:209 +0x159 github.com/cockroachdb/cockroach/pkg/kv.(*txnSeqNumAllocator).SendLocked(0xc005c14268, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_seq_num_allocator.go:89 +0x226 github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).SendLocked(0xc005c141c8, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:168 +0x1a6 github.com/cockroachdb/cockroach/pkg/kv.(*TxnCoordSender).Send(0xc005c14000, 0x489e1a0, 0xc006eb3c20, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_coord_sender.go:800 +0x46d github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc000156080, 0x489e1a0, 0xc006eb3680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Send(0xc005f75b00, 0x489e1a0, 0xc006eb3680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:807 +0x13c github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e1a0, 0xc006eb3680, 0xc0029b0bb8, 0xc00327a100, 0x8, 0xc003acb4f0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Run(0xc005f75b00, 0x489e1a0, 0xc006eb3680, 0xc00327a100, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:500 +0xd8 github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).Get(0xc005f75b00, 0x489e1a0, 0xc006eb3680, 0x3d0b580, 0xc0039072c0, 0x1258fac, 0x117cf56, 0xc004f1c910, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:312 +0xbd github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*TableDescriptor).validateCrossReferences(0xc002ec6700, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:1534 +0xa8 github.com/cockroachdb/cockroach/pkg/sql/sqlbase.(*TableDescriptor).Validate(0xc002ec6700, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc005f75b00, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/sqlbase/structured.go:1526 +0x86 github.com/cockroachdb/cockroach/pkg/sql.getDescriptorByID(0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc, 0x4913d60, 0xc0029b1bd8, 0x1, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/descriptor.go:183 +0x299 github.com/cockroachdb/cockroach/pkg/sql.UncachedPhysicalAccessor.GetObjectDesc(0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc0049358f8, 0x0, 0x30, 0x3d14180, 0x1, 0xc006eb3710) /go/src/github.com/cockroachdb/cockroach/pkg/sql/physical_schema_accessors.go:160 +0x347 github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getTableVersion.func1(0x3d14180, 0xc006eb3710, 0xc0049358f8) /go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:257 +0x6d github.com/cockroachdb/cockroach/pkg/sql.(*TableCollection).getTableVersion(0xc004935238, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc0049358f8, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/table.go:265 +0x9d5 github.com/cockroachdb/cockroach/pkg/sql.(*CachedPhysicalAccessor).GetObjectDesc(0xc003906960, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc0049358f8, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/physical_schema_accessors.go:238 +0x15f github.com/cockroachdb/cockroach/pkg/sql.(*LogicalSchemaAccessor).GetObjectDesc(0xc0039069a0, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc0049358f8, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/logical_schema_accessors.go:93 +0x652 github.com/cockroachdb/cockroach/pkg/sql.(*planner).LookupObject(0xc004935358, 0x489e1a0, 0xc006eb3680, 0x1, 0x404759d, 0x6, 0x3f7e8f6, 0x6, 0x40475a4, 0x8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:292 +0x183 github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*TableName).ResolveExisting(0xc004935b10, 0x489e1a0, 0xc006eb3680, 0x7f6d0704cd48, 0xc004935358, 0x1, 0x3f7ec9e, 0x6, 0xc003ac4430, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/sem/tree/name_resolution.go:297 +0x24e github.com/cockroachdb/cockroach/pkg/sql.resolveExistingObjectImpl(0x489e1a0, 0xc006eb3680, 0x48fe360, 0xc004935358, 0xc004935b10, 0x7f6d00000001, 0x0, 0xc003c6efc8, 0x13, 0x727d960, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:140 +0x182 github.com/cockroachdb/cockroach/pkg/sql.ResolveExistingObject(0x489e1a0, 0xc006eb3680, 0x48fe360, 0xc004935358, 0xc004935b10, 0xc000000001, 0x0, 0xc006eb36b0, 0x30, 0x30) /go/src/github.com/cockroachdb/cockroach/pkg/sql/resolver.go:101 +0x9a github.com/cockroachdb/cockroach/pkg/sql.(*optCatalog).ResolveDataSource(0xc004935af8, 0x489e1a0, 0xc006eb3680, 0xc003260000, 0xc0014ec4f8, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt_catalog.go:170 +0xf1 github.com/cockroachdb/cockroach/pkg/sql/opt.(*Metadata).CheckDependencies(0xc0009c9040, 0x489e1a0, 0xc006eb3680, 0x48fa040, 0xc004935af8, 0x6fb2f60, 0xc000d23c01, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/metadata.go:220 +0x359 github.com/cockroachdb/cockroach/pkg/sql/opt/memo.(*Memo).IsStale(0xc0009c9040, 0x489e1a0, 0xc006eb3680, 0xc004935428, 0x48fa040, 0xc004935af8, 0xc0009c9001, 0xc000fb2f00, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/opt/memo/memo.go:279 +0xbb github.com/cockroachdb/cockroach/pkg/sql.(*planner).prepareUsingOptimizer(0xc004935358, 0x489e1a0, 0xc006eb3680, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/plan_opt.go:83 +0x7bf github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).populatePrepared(0xc004935000, 0x489e1a0, 0xc006eb3680, 0xc005f75b00, 0xc003906ac0, 0x4, 0x4, 0xc004935358, 0x1, 0xa8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:204 +0x169 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).prepare(0xc004935000, 0x489e1a0, 0xc006eb3680, 0x48c22e0, 0xc0026dbf90, 0x4047591, 0x7b, 0x4, 0x1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:159 +0x442 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).addPreparedStmt(0xc004935000, 0x489e1a0, 0xc006eb3680, 0x0, 0x0, 0x48c22e0, 0xc0026dbf90, 0x4047591, 0x7b, 0x4, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:97 +0xf3 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPrepare(0xc004935000, 0x489e1a0, 0xc006eb3680, 0x0, 0x0, 0x48c22e0, 0xc0026dbf90, 0x4047591, 0x7b, 0x4, ...) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor_prepare.go:51 +0x171 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd(0xc004935000, 0x489e0e0, 0xc007629c00, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1308 +0x22b2 github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run(0xc004935000, 0x489e1a0, 0xc006eb32c0, 0xc00022cc80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1167 +0x1a3 github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx.func1(0xc004935000, 0x489e1a0, 0xc006eb32c0, 0xc000c36d80, 0xc003906800, 0xc005fc9950, 0xc003acb150) /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:201 +0x64 created by github.com/cockroachdb/cockroach/pkg/sql.(*internalExecutorImpl).initConnEx /go/src/github.com/cockroachdb/cockroach/pkg/sql/internal.go:200 +0x4b9 goroutine 238747 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c229258, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc000f7a818, 0x72, 0x4100, 0x4114, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc000f7a800, 0xc002cd8800, 0x4114, 0x4114, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc000f7a800, 0xc002cd8800, 0x4114, 0x4114, 0x203000, 0x0, 0x4107) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f94000, 0xc002cd8800, 0x4114, 0x4114, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc003634180, 0xc002cd8800, 0x4114, 0x4114, 0x380, 0x373, 0xc0043bb0e8) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc000c7e958, 0x482c340, 0xc003634180, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc000c7e700, 0x48317e0, 0xc000f94000, 0x5, 0xc000f94000, 0xc00008c000) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc000c7e700, 0x40e5100, 0xc000c7e838, 0x37) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc000c7e700, 0xc001a9c000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc00069f7a0, 0xc001708b20, 0x5, 0x200, 0x40e5218, 0xc0043bb420, 0x18) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc00069f7a0, 0xc001708b20, 0x5, 0x200, 0x5, 0x1a, 0x0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recvMessage(0xc001708b00, 0xc0043bb580, 0x1a, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:929 +0x104 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1Buf(0xc001708b00, 0xc0043bb580, 0x15) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:978 +0x39 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).recv1(...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:999 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).simpleExec(0xc001708b00, 0x3f9e49a, 0x14, 0xc0043bb668, 0x0, 0x736ebc, 0x0, 0x3aaaa20, 0x1) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:603 +0x1ee github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).Exec(0xc001708b00, 0x3f9e49a, 0x14, 0x72b56a8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn.go:857 +0x247 github.com/cockroachdb/cockroach/vendor/github.com/lib/pq.(*conn).ExecContext(0xc001708b00, 0x489e120, 0xc00010c010, 0x3f9e49a, 0x14, 0x72b56a8, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/lib/pq/conn_go18.go:42 +0x195 database/sql.ctxDriverExec(0x489e120, 0xc00010c010, 0x7f6d05dc2668, 0xc001708b00, 0x0, 0x0, 0x3f9e49a, 0x14, 0x72b56a8, 0x0, ...) /usr/local/go/src/database/sql/ctxutil.go:31 +0x233 database/sql.(*DB).execDC.func2() /usr/local/go/src/database/sql/sql.go:1467 +0x1db database/sql.withLock(0x4859960, 0xc001022b00, 0xc0043bb9b8) /usr/local/go/src/database/sql/sql.go:3097 +0x63 database/sql.(*DB).execDC(0xc0017075c0, 0x489e120, 0xc00010c010, 0xc001022b00, 0xc0043bbab8, 0x3f9e49a, 0x14, 0x0, 0x0, 0x0, ...) /usr/local/go/src/database/sql/sql.go:1462 +0x47d database/sql.(*DB).exec(0xc0017075c0, 0x489e120, 0xc00010c010, 0x3f9e49a, 0x14, 0x0, 0x0, 0x0, 0x40e5101, 0x33, ...) /usr/local/go/src/database/sql/sql.go:1447 +0x139 database/sql.(*DB).ExecContext(0xc0017075c0, 0x489e120, 0xc00010c010, 0x3f9e49a, 0x14, 0x0, 0x0, 0x0, 0x485f9e0, 0xc003634160, ...) /usr/local/go/src/database/sql/sql.go:1425 +0xde github.com/cockroachdb/cockroach/pkg/testutils/sqlutils.(*SQLRunner).Exec(0xc00492c350, 0x49240c0, 0xc0073ed400, 0x3f9e49a, 0x14, 0x0, 0x0, 0x0, 0x485f9e0, 0xc003634160) /go/src/github.com/cockroachdb/cockroach/pkg/testutils/sqlutils/sql_runner.go:52 +0xbc github.com/cockroachdb/cockroach/pkg/ccl/partitionccl.TestRepartitioning.func1(0xc0073ed400) /go/src/github.com/cockroachdb/cockroach/pkg/ccl/partitionccl/partition_test.go:1293 +0x103 testing.tRunner(0xc0073ed400, 0xc004a043f0) /usr/local/go/src/testing/testing.go:865 +0xc0 created by testing.(*T).Run /usr/local/go/src/testing/testing.go:916 +0x35a goroutine 238978 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc004d98200, 0xc00601eb29, 0xc001edd298) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0068a8300, 0x3e99440, 0xc0051e7740, 0xc001edd2f0, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0068a8300, 0xc0044811c0, 0x3c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00658efc0, 0xc001edd438, 0xc001edd428, 0xc0049c2660, 0xc00601eb29) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00658efc0, 0x3e99440, 0xc0051e7740, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc0051e6cc0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0068a8280, 0x3e99440, 0xc0051e7740, 0xc002d8d880, 0xc0019c0640, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc0051e6cc0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0068a8280, 0x3e99440, 0xc0051e7740, 0xc002d8d880, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc002d8d880, 0x489e160, 0xc0051e6cc0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0068a8280, 0x3e99440, 0xc0051e7740, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc0001107e8, 0x489e160, 0xc0051e6cc0, 0xc0068a8280, 0x0, 0x0, 0x0, 0x1, 0x489e160, 0xc0051e6cc0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006f096c0, 0x489e160, 0xc0051e6cc0, 0x1, 0x485a560, 0xc0001107e8, 0x0, 0x0, 0x100000001, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006f096c0, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x100000001, 0x1, 0xc00601e830, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc001c55560, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc001d92960, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc001d92900, 0x489e160, 0xc0051e6cc0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e160, 0xc0051e6cc0, 0xc001edf3a8, 0xc0021a2100, 0x8, 0xc00601ea48) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc001d92900, 0x489e160, 0xc0051e6cc0, 0xc0021a2100, 0x2, 0x2) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminRelocateRange(0xc001d92900, 0x489e160, 0xc0051e6cc0, 0x3cd5280, 0xc005ac1340, 0xc00601ead0, 0x2, 0x2, 0xc006420af0, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:605 +0xa5 github.com/cockroachdb/cockroach/pkg/storage.(*mergeQueue).process(0xc0002c8fa0, 0x489e160, 0xc0051e6cc0, 0xc00579d000, 0xc007545ae0, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/merge_queue.go:341 +0xda3 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc0051e6cc0, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc0051e6cc0, 0xc006422a60, 0x1f, 0xdf8475800, 0xc004baa930, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc001682780, 0x489e1a0, 0xc004baa900, 0x4906de0, 0xc00579d000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc004baa600) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc004baa600, 0xc0049c24e0, 0x29, 0x0, 0x0, 0xc004baa840) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 238968 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc0067cd2c0, 0x0, 0x1, 0xc001bd4120, 0xfaa) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc0, 0xc0051a8000, 0xc0000aef50, 0x254e946) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc004480a00, 0x33, 0x0, 0x0, 0xc003ca2960) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6388 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003c099a0, 0xc003aea510, 0x5, 0x5, 0x0, 0x0, 0xc003966c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003c099a0, 0xc003aea510, 0x5, 0x5, 0xc11b10, 0xc000357a60, 0xc0038d5af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003af1230, 0xc003aea510, 0x5, 0x5, 0x3c, 0xc0038d5b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003af1230, 0xc003aea510, 0x5, 0x5, 0x5, 0xc0038d5bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002d1e400, 0xc003aea510, 0x5, 0x5, 0x203000, 0x203000, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003aea500, 0x7fffffff, 0x732a2f, 0xc002ec9500, 0x4, 0xc0038d5c70, 0x733ecb, 0xc001dfdb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003aea500, 0xc002d1e400, 0x0, 0x0, 0x7fffffff, 0xc0038d5e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003aea500, 0x7f6d0c1990d0, 0xc000292db0, 0xc002d1e400, 0x0, 0x0, 0x3de86e0, 0xc0036ade80, 0x7fffffff, 0xc0038d5e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00241ccc0, 0x3de86e0, 0xc0036ade80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc000d57f90, 0x489e1a0, 0xc003af12f0, 0xc003966c00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc000d57f90, 0xc0018a82d0, 0x489e1a0, 0xc003af12f0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003af12f0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57fe0, 0xc0019e4510, 0xc003af12c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 239531 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc002d50900, 0x0, 0x1, 0x7, 0xc002238a20) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6, 0xc006a2f8c0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00776d2c0, 0x33, 0x0, 0x0, 0xc0051ff200) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6387 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc0018a82d0, 0x48fa580, 0xc000d57f90, 0x485c120, 0xc0018a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00241ccc0, 0x0, 0xc001b067e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00241ccc0, 0xc003aea520, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc00241ccc0, 0xc003aea520, 0x40e1ba8, 0x489e1a0, 0xc003af1290) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc002d1e400, 0xc0009a98f0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc002d1e400, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003f2580, 0xc00262b200, 0x48fef60, 0xc0014f0780, 0xc002d1e400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 236178 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc000301880, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc0026af080, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc001af5c70, 0x489e120, 0xc00010c010, 0xc0, 0xc00740b0e0, 0xc001858f38, 0xc00685f0a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e120, 0xc00010c010, 0xc003ede940, 0x33, 0x0, 0x0, 0xc004d4e1b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 9964 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc001460780, 0x3, 0xc003bb1020, 0xc00407cbc0, 0x48fa520, 0xc001c0e600, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001799b60) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001c0e5a0, 0xc000937c20, 0xc001799b30) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 5686 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc00461f140, 0xc0002342d0, 0xf, 0xc00463d440, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc003f0ba40) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0045aad90, 0xc000937c20, 0xc004922d70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6322 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc003c097c0, 0xc003aea330, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003c097c0, 0xc003aea330, 0x5, 0x5, 0xc11b10, 0xc0003579c0, 0xc00375b978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003af1020, 0xc003aea330, 0x5, 0x5, 0x74cff0, 0xc00375b9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003af1020, 0xc003aea330, 0x5, 0x5, 0x5, 0xf, 0xc00107e1e0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc002d1e300, 0xc003aea330, 0x5, 0x5, 0x713a00, 0xc00624f670, 0xc00375ba78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003aea320, 0x7fffffff, 0x3c06120, 0xc00107e1e0, 0xc00375ba0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003aea320, 0xc002d1e300, 0x0, 0x0, 0x7fffffff, 0xc00375bcb8, 0x4886820, 0x72b56a8, 0x0, 0xc002c18840, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003aea320, 0x7f6d0c1990d0, 0xc000292db0, 0xc002d1e300, 0x0, 0x0, 0x3de8800, 0xc004c99900, 0x7fffffff, 0xc00375bcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0033bd980, 0x3de8800, 0xc004c99900, 0xc00375bcb8, 0x2, 0xc004c99900) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0033bd980, 0xc000080570, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc00073e5a0, 0xc00375be00, 0xc00375bdf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc00073e5a0, 0x3de8800, 0xc004c99900, 0x2c5e8e0, 0xc000932000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc000d57e30, 0x489e1a0, 0xc003af10b0, 0xc0064f5700) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc000d57e30, 0xc002a4dd00, 0xc001460780, 0x489e1a0, 0xc003af10b0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003af10b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57e70, 0xc000937c20, 0xc003af1050) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 119428 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).addToPurgatoryLocked.func2(0x489e1a0, 0xc0052e38c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:1086 +0x325 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00161be90, 0xc0019e4510, 0xc00395ee00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 10234 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc003a0bcc0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc002a59680, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc00492e700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 236026 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc004185740, 0x0, 0x1, 0x38, 0x50) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc0, 0xc004885200, 0xc0014ea000, 0x15c106cd307a3107) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc004bd2c80, 0x33, 0x0, 0x0, 0xc00586c930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 235229 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc003932480, 0x0, 0x1, 0xc0000af5e0, 0x7ee93c) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc00532a7e0, 0xc003c37000, 0x15c106cd26cbc383) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0055d4b80, 0x33, 0x0, 0x0, 0xc0064b7a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10232 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc00492e700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 21221 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc0040aced0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc0037973d0, 0xc000937c20, 0xc0037973c0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6810 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc001a22040) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 6315 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc003c08eb0, 0xc003c97310, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003c08eb0, 0xc003c97310, 0x5, 0x5, 0xc11b10, 0xc0003571a0, 0xc004405978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003bd7a10, 0xc003c97310, 0x5, 0x5, 0x74cff0, 0xc0044059a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003bd7a10, 0xc003c97310, 0x5, 0x5, 0x5, 0xf, 0xc0011f5620, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0037f7900, 0xc003c97310, 0x5, 0x5, 0x83e500, 0xc004f60830, 0xc004405a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003c97300, 0x7fffffff, 0x3c06120, 0xc0011f5620, 0xc004405a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003c97300, 0xc0037f7900, 0x0, 0x0, 0x7fffffff, 0xc004405cb8, 0x4886820, 0x72b56a8, 0x0, 0xc005fb1260, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003c97300, 0x7f6d0c1990d0, 0xc000292db0, 0xc0037f7900, 0x0, 0x0, 0x3de8800, 0xc002604b80, 0x7fffffff, 0xc004405cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0033bce80, 0x3de8800, 0xc002604b80, 0xc004405cb8, 0xc003392ff0, 0xc002604b80) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0033bce80, 0xc00009ea70, 0x203000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0024e70e0, 0xc004405e00, 0xc004405df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0024e70e0, 0x3de8800, 0xc002604b80, 0x2c5e8e0, 0xc00084c700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc000d57410, 0x489e1a0, 0xc003bd7aa0, 0xc006011b80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc000d57410, 0xc002a4dbc0, 0xc0018a82d0, 0x489e1a0, 0xc003bd7aa0, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc003bd7aa0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57470, 0xc0019e4510, 0xc003bd7a40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 135446 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc000c3d500) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e1a0, 0xc000c3d500, 0xc00435ab10, 0x2a, 0x0, 0x0, 0xc00156bb90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 10251 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc002f6a720, 0xc001ae61a0, 0xf, 0xc00367d260, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc003d0ae40) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003d3d330, 0xc0019e4510, 0xc004108f50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 238655 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc000301880, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc00781f1a0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc001af5c70, 0x489e120, 0xc00010c010, 0xc0, 0xc00709a7e0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e120, 0xc00010c010, 0xc006c97940, 0x33, 0x0, 0x0, 0xc00597ea50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 239649 [semacquire]: sync.runtime_SemacquireMutex(0x726cb6c, 0xc919fc400) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0x726cb68) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000002, 0x6bde2b1, 0x1a, 0x77d, 0xc0044f05a0, 0x99) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:857 +0x206 github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc007160ba0, 0xc000000002, 0x2, 0x3fbb574, 0x1d, 0xc00376dcb0, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc007160ba0, 0x1, 0x2, 0x3fbb574, 0x1d, 0xc00376dcb0, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Warningf(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:129 github.com/cockroachdb/cockroach/pkg/storage.(*Store).AdminRelocateRange.func2() /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1917 +0x140 github.com/cockroachdb/cockroach/pkg/storage.(*Store).AdminRelocateRange(0xc000932000, 0x489e1a0, 0xc007160ba0, 0xc1, 0xc003de7df0, 0x3, 0x8, 0xc003de7df8, 0x3, 0x8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:2026 +0xa52 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc0035be800, 0x489e1a0, 0xc007160ba0, 0xc006a4f480, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1335 +0x803 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc0035be800, 0x489e1a0, 0xc007160b70, 0xc1, 0x15c106cd54860ebf, 0x0, 0x200000002, 0x3, 0xc006417fc0, 0xc1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc0035be800, 0x489e1a0, 0xc007160b70, 0x15c106cd54860ebf, 0x0, 0x200000002, 0x3, 0xc006417fc0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000932000, 0x489e1a0, 0xc007160b10, 0x15c106cd54860ebf, 0x0, 0x200000002, 0x3, 0xc006417fc0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000f63cc0, 0x489e1a0, 0xc007160b10, 0x0, 0x0, 0x200000002, 0x3, 0xc006417fc0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc007160ae0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000937c20, 0x489e1a0, 0xc007160ae0, 0x3f956b7, 0x10, 0xc00376f710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00174a000, 0x489e1a0, 0xc007160ae0, 0xc006a4f400, 0xc007160ae0, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00174a000, 0x489e1a0, 0xc007160ab0, 0xc006a4f400, 0xc00174a000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc007160ab0, 0x3f4fce0, 0xc006a4f400, 0x3f4fce0, 0xc006a4f400, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc007160ab0, 0x3f4fce0, 0xc006a4f400, 0xc005e8be60, 0xc005e8be80, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc007160ab0, 0x3f4fce0, 0xc006a4f400, 0xc005e8be60, 0xc005e8be80, 0x6ff238, 0x20, 0x3b5a920, 0xc006b1ef01) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc007160ab0, 0x3f4fce0, 0xc006a4f400, 0xc005e8be60, 0xc005e8be80, 0xc002be2bc8, 0x7edeba, 0x3d2f600, 0xc007160ab0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc00174a000, 0x489e1a0, 0xc007160ab0, 0xc006b1ef00, 0xc000d57350, 0x489e1a0, 0xc007160ab0, 0xc002520c00, 0x25) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00193c600, 0x48fef60, 0xc001892180, 0xc0072b8d00, 0xc00111ba10, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc001892180, 0xc0072b8d00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc000318e10, 0xc00193c600, 0x48fef60, 0xc001892180, 0xc0072b8d00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 7551 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc0033c9f80, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc003abbce0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc000367500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 238863 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc0009f2e00, 0xc005284b40, 0x489e1a0, 0xc00628c8d0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 236122 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc4, 0x490bae0, 0xc004aa4d20, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc4, 0xc004149d40, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003f6dbc0, 0x33, 0x0, 0x0, 0xc003256810) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7025 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc0034a40f0, 0xc00367e170, 0x5, 0x5, 0x0, 0x0, 0xc004813000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0034a40f0, 0xc00367e170, 0x5, 0x5, 0xc11b10, 0xc0018f6080, 0xc00496daf8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc002f6e180, 0xc00367e170, 0x5, 0x5, 0x3b, 0xc00496db20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc002f6e180, 0xc00367e170, 0x5, 0x5, 0x5, 0xc00496dbd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc000c26000, 0xc00367e170, 0x5, 0x5, 0x203001, 0x203001, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00367e160, 0x7fffffff, 0x732a2f, 0xc001745e00, 0x4, 0xc00496dc70, 0x733ecb, 0xc0024278c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00367e160, 0xc000c26000, 0x0, 0x0, 0x7fffffff, 0xc00496de30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00367e160, 0x7f6d0c1990d0, 0xc000292db0, 0xc000c26000, 0x0, 0x0, 0x3de86e0, 0xc0077a2720, 0x7fffffff, 0xc00496de30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc003366000, 0x3de86e0, 0xc0077a2720, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc00403e0b0, 0x489e1a0, 0xc002f6e240, 0xc004813000) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc00403e0b0, 0xc001460780, 0x489e1a0, 0xc002f6e240, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc002f6e240) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc00403e0d0, 0xc000937c20, 0xc002f6e210) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7550 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce3f8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0016bf218, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0016bf200, 0xc003592000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0016bf200, 0xc003592000, 0x791, 0x791, 0x203001, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000110ca8, 0xc003592000, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc005f81fc0, 0xc003592000, 0x791, 0x791, 0x14e, 0x482c600, 0xc0045a2a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc003084258, 0x482c340, 0xc005f81fc0, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc003084000, 0x48317e0, 0xc000110ca8, 0x5, 0xc000110ca8, 0xc004bab680) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc003084000, 0x40e5100, 0xc003084138, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc003084000, 0xc0042ac000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003abbc20, 0xc003442c78, 0x9, 0x9, 0xc007a3bc18, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003abbc20, 0xc003442c78, 0x9, 0x9, 0x9, 0xc004baac60, 0x485e320, 0xc00601e7f0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc003442c78, 0x9, 0x9, 0x482c140, 0xc003abbc20, 0x0, 0xc000000000, 0xc0079f7b00, 0x4bab600) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc003442c40, 0xc004bab650, 0xc004bab650, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc000367500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 239073 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc0072b8c00, 0xc006417f98, 0xc00256d298) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc006a4f380, 0x3e99440, 0xc006b1ecc0, 0xc00256d2f0, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc006a4f380, 0xc006bdcd00, 0x37) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc005570120, 0xc00256d438, 0xc00256d428, 0xc0038f3b90, 0xc006417f98) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc005570120, 0x3e99440, 0xc006b1ecc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc003cca1e0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc006a4f300, 0x3e99440, 0xc006b1ecc0, 0xc0015a6e00, 0xc0015f27c0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc003cca1e0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc006a4f300, 0x3e99440, 0xc006b1ecc0, 0xc0015a6e00, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc0015a6e00, 0x489e160, 0xc003cca1e0, 0x3fc5646, 0x21, 0x3f4fce0, 0xc006a4f300, 0x3e99440, 0xc006b1ecc0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc00475c340, 0x489e160, 0xc003cca1e0, 0xc006a4f300, 0x0, 0x0, 0x0, 0x2, 0x489e160, 0xc003cca1e0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc00667c9c0, 0x489e160, 0xc003cca1e0, 0x2, 0x485a560, 0xc00475c340, 0x0, 0x0, 0x200000002, 0x3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc00667c9c0, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x200000002, 0x3, 0xc005e23304, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000c42000, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0001560e0, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc000156080, 0x489e160, 0xc003cca1e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e160, 0xc003cca1e0, 0xc00256f3a8, 0xc00645d600, 0x8, 0xc006417ef0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc000156080, 0x489e160, 0xc003cca1e0, 0xc00645d600, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminRelocateRange(0xc000156080, 0x489e160, 0xc003cca1e0, 0x3cd5280, 0xc005e8ba40, 0xc006417f58, 0x1, 0x1, 0xc0003c6fb0, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:605 +0xa5 github.com/cockroachdb/cockroach/pkg/storage.(*mergeQueue).process(0xc0002f4060, 0x489e160, 0xc003cca1e0, 0xc00511b800, 0xc0053f4af0, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/merge_queue.go:341 +0xda3 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc003cca1e0, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc003cca1e0, 0xc00255e840, 0x1f, 0xdf8475800, 0xc005130cc0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc000158b40, 0x489e1a0, 0xc005130c90, 0x4906de0, 0xc00511b800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc005130c30) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e1a0, 0xc005130c30, 0xc006398360, 0x29, 0x0, 0x0, 0xc005130c60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 5802 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc004c95480) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 236118 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc0075998c0, 0x0, 0x1, 0xc004772c00, 0x40) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc1, 0xc003c9cea0, 0xc00470f000, 0x15c106cd3328a3aa) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003f6d3c0, 0x33, 0x0, 0x0, 0xc003d94120) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7715 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0038681e0, 0xc00180e610, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0038681e0, 0xc00180e610, 0x5, 0x5, 0xc11b10, 0xc001d10120, 0xc004ac7978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0015831a0, 0xc00180e610, 0x5, 0x5, 0x74cff0, 0xc004ac79a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc0015831a0, 0xc00180e610, 0x5, 0x5, 0x5, 0xf, 0xc0011f5620, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0010f0000, 0xc00180e610, 0x5, 0x5, 0x83e500, 0xc0038cb720, 0xc004ac7a78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc00180e600, 0x7fffffff, 0x3c06120, 0xc0011f5620, 0xc004ac7a0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc00180e600, 0xc0010f0000, 0x0, 0x0, 0x7fffffff, 0xc004ac7cb8, 0x4886820, 0x72b56a8, 0x0, 0xc0050da400, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc00180e600, 0x7f6d0c1990d0, 0xc000292db0, 0xc0010f0000, 0x0, 0x0, 0x3de8800, 0xc005ebd940, 0x7fffffff, 0xc004ac7cb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0041e6000, 0x3de8800, 0xc005ebd940, 0xc004ac7cb8, 0xc0029a3bd0, 0xc005ebd940) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0041e6000, 0xc0000a0f70, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc002f8a000, 0xc004ac7e00, 0xc004ac7df0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc002f8a000, 0x3de8800, 0xc005ebd940, 0x2c5e8e0, 0xc00084c700) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc002aa20e0, 0x489e1a0, 0xc001583230, 0xc0048424c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc002aa20e0, 0xc0046b6040, 0xc0018a82d0, 0x489e1a0, 0xc001583230, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc001583230) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc002aa2100, 0xc0019e4510, 0xc0015831d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 5806 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004c95500, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00459c120, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001084540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 5803 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc0008d2680, 0xc0037c7d40, 0xc00028c7b0, 0xf, 0xc003f12300, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc00400e3f0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc004120850, 0xc000937c20, 0xc002d86f00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 236765 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6e, 0x490bae0, 0xc0026390e0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6e, 0xc007020b40, 0xc005af8000, 0x15c106cd3e161339) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003925640, 0x33, 0x0, 0x0, 0xc002d0e480) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6314 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream.func5(0xc001ce2000, 0xc0024e70e0, 0x489e1a0, 0xc003bd78f0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:320 +0xd7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newClientStream /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:319 +0x991 goroutine 235624 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6a, 0x490bae0, 0xc0072da000, 0x0, 0x1, 0xc001475e18, 0x2b9744f) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6a, 0xc00786b320, 0xc000944000, 0x15c106cd2a33b765) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc002968000, 0x33, 0x0, 0x0, 0xc00524e000) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 5804 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001084540) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 238818 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc2, 0x490bae0, 0xc0076605a0, 0x0, 0x1, 0x7, 0xc0038f2db0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc2, 0xc0051747e0, 0xc006c1d9e0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00776cb00, 0x33, 0x0, 0x0, 0xc0074dcb70) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 235761 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6c, 0x490bae0, 0xc007d61020, 0x0, 0x1, 0xc004070e00, 0x26) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6c, 0xc005f359e0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0003a7c40, 0x33, 0x0, 0x0, 0xc004fe5620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 5822 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc000a25600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 235469 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc0051e66c0, 0x0, 0x1, 0xc000b512e8, 0xc00189fe40) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc000c438c0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0042fb0c0, 0x33, 0x0, 0x0, 0xc0045bb3b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 3746 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc0041ae7b0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003dae1a0, 0xc000a30090, 0xc003dae190) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 239291 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc3, 0x490bae0, 0xc006844180, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc3, 0xc005284900, 0xc0014ea000, 0x15c106cd506945f5) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc006692640, 0x33, 0x0, 0x0, 0xc00626f5f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 240094 [select]: github.com/cockroachdb/cockroach/pkg/kv.(*txnHeartbeater).heartbeatLoop(0xc001f433c8, 0x489e0e0, 0xc007629640) /go/src/github.com/cockroachdb/cockroach/pkg/kv/txn_interceptor_heartbeater.go:244 +0x165 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e0e0, 0xc007629640, 0xc002a309c0, 0x29, 0x4913e60, 0xc000364d20, 0xc003ac4290) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6149 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce4c8, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc001952498, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc001952480, 0xc00478b800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc001952480, 0xc00478b800, 0x75b, 0x75b, 0xc00495e938, 0xc005ae0401, 0x972cfaec0204b2e2) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc000f94108, 0xc00478b800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0039205a0, 0xc00478b800, 0x75b, 0x75b, 0xc004bb7930, 0x1, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc004a143e0, 0xc00478b800, 0x75b, 0x75b, 0x203000, 0x140, 0xc00495ea50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0023e9058, 0x482c340, 0xc004a143e0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0023e8e00, 0x7f6d0700c008, 0xc0039205a0, 0x5, 0xc0039205a0, 0x0) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0023e8e00, 0x40e5100, 0xc0023e8f38, 0xc005b84870) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0023e8e00, 0xc002ff0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003851800, 0xc00319e2d8, 0x9, 0x9, 0xc004c25948, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003851800, 0xc00319e2d8, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00319e2d8, 0x9, 0x9, 0x482c140, 0xc003851800, 0x0, 0xc500000000, 0xc006b60000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00319e2a0, 0xc00404d920, 0xc00404d920, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003e00780, 0xc0041c3590, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc003e00780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc003e00780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 6148 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003e00780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 6147 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004c61400, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc0047a6240, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003e00780) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 6020 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce738, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc0041e6618, 0x72, 0x700, 0x791, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc0041e6600, 0xc00478a800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc0041e6600, 0xc00478a800, 0x791, 0x791, 0x203000, 0x0, 0x784) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc003316d38, 0xc00478a800, 0x791, 0x791, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 crypto/tls.(*atLeastReader).Read(0xc003c69b20, 0xc00478a800, 0x791, 0x791, 0x6e, 0x482c600, 0xc001a53a78) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0023e8cd8, 0x482c340, 0xc003c69b20, 0x6fcab5, 0x3aee1e0, 0x3ee1be0) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0023e8a80, 0x48317e0, 0xc003316d38, 0x5, 0xc003316d38, 0xc005867320) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0023e8a80, 0x40e5100, 0xc0023e8bb8, 0x6f833c) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0023e8a80, 0xc0026dc000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc004200720, 0xc003b9c038, 0x9, 0x9, 0xc00804ccb8, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc004200720, 0xc003b9c038, 0x9, 0x9, 0x9, 0xc002c55bf0, 0x485e320, 0xc003a1f6b0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc003b9c038, 0x9, 0x9, 0x482c140, 0xc004200720, 0x0, 0xc000000000, 0xc007f03500, 0x5867200) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc003b9c000, 0xc0058672f0, 0xc0058672f0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc0041ba000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1237 +0x168 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:286 +0xd15 goroutine 6021 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004932940, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004200840, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc0041ba000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:330 +0x7b created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:328 +0xeb2 goroutine 237663 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc000301880, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc005954960, 0x0, 0x1, 0xc005b0e0c0, 0xc001b76618) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc001af5c70, 0x489e120, 0xc00010c010, 0xc0, 0xc003e897a0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e120, 0xc00010c010, 0xc006bdd000, 0x33, 0x0, 0x0, 0xc006c40ae0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 34420 [select]: github.com/cockroachdb/cockroach/pkg/storage/idalloc.(*Allocator).start.func1(0x489e1a0, 0xc004c27e00) /go/src/github.com/cockroachdb/cockroach/pkg/storage/idalloc/id_alloc.go:123 +0x436 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc003d183c0, 0xc0019e4510, 0xc003d183a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 7594 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc001ece180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 6313 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0018a82d0, 0x2, 0xc003ee9200, 0xc002a4dbc0, 0x48fa520, 0xc000d57410, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003bd7830) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d572f0, 0xc0019e4510, 0xc003bd77d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 236119 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc4, 0x490bae0, 0xc007599e60, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc4, 0xc005d74c60, 0xc003a96000, 0x15c106cd352ffdf2) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003f6d6c0, 0x33, 0x0, 0x0, 0xc003d95650) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6037 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce668, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc003847298, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003847280, 0xc00478b000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc003847280, 0xc00478b000, 0x75b, 0x75b, 0xc0024eb938, 0x701, 0xc0023e8880) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc003316ce8, 0xc00478b000, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc0038507e0, 0xc00478b000, 0x75b, 0x75b, 0xc00478b00d, 0x74, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0036adc80, 0xc00478b000, 0x75b, 0x75b, 0x91, 0x482c600, 0xc0024eba50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc0023e8958, 0x482c340, 0xc0036adc80, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc0023e8700, 0x7f6d0700c008, 0xc0038507e0, 0x5, 0xc0038507e0, 0x74) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc0023e8700, 0x40e5100, 0xc0023e8838, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc0023e8700, 0xc0026c0000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc003851560, 0xc00319e118, 0x9, 0x9, 0xc001119bc8, 0xc0046bddc0, 0x6b) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc003851560, 0xc00319e118, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc00319e118, 0x9, 0x9, 0x482c140, 0xc003851560, 0x0, 0x300000000, 0xc0017d2100, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc00319e0e0, 0xc00404d440, 0xc00404d440, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc003e00600, 0xc002e43b90, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00262b200, 0x48fef60, 0xc003e00600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00262b200, 0x48fef60, 0xc003e00600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 238972 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00174c380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc0067cda40, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc00215c8f0, 0x489e120, 0xc00010c010, 0xc1, 0xc004d3cb40, 0xc000480f38, 0xc007c87960) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e120, 0xc00010c010, 0xc004480c80, 0x33, 0x0, 0x0, 0xc003ca39b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6035 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc004c612c0, 0x1, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:317 +0x104 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc004200b40, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/controlbuf.go:435 +0x1b6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server.func2(0xc003e00600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:276 +0xcb created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:273 +0xfba goroutine 6036 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).keepalive(0xc003e00600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:935 +0x1ed created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Server /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:282 +0xfdf goroutine 7549 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc000367500) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:1288 +0x113 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.newHTTP2Client /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_client.go:281 +0x1666 goroutine 237955 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00174c380, 0x489e120, 0xc00010c010, 0xc1, 0x490bae0, 0xc007c2b860, 0x0, 0x1, 0xc000ee3ec0, 0xbab23ccf) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc00215c8f0, 0x489e120, 0xc00010c010, 0xc1, 0xc007a9be60, 0xc0004aa800, 0x15c106cd4a1feba7) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e120, 0xc00010c010, 0xc004480540, 0x33, 0x0, 0x0, 0xc0071426f0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7024 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc001460780, 0x48fa580, 0xc00403e0b0, 0x485c120, 0xc001460780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc003366000, 0x0, 0xc001b07200) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc003366000, 0xc00367e180, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc003366000, 0xc00367e180, 0x40e1ba8, 0x489e1a0, 0xc002f6e1e0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc001ece180, 0xc000c26000, 0xc00111b2c0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc001ece180, 0xc000c26000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc003f85710, 0xc00193c600, 0x48fef60, 0xc001ece180, 0xc000c26000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 6812 [chan receive]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).resetTransport(0xc00276c2c0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:1107 +0x5a7 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*addrConn).connect /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/clientconn.go:767 +0xb6 goroutine 6320 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc001460780, 0x3, 0xc003ee9bc0, 0xc002a4dd00, 0x48fa520, 0xc000d57e30, 0x48fa500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc003af0e40) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d57d90, 0xc000937c20, 0xc003af0de0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 235965 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc0037edb60, 0x0, 0x1, 0xc0055d1e00, 0x2e) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc004dd30e0, 0xc00425e800, 0x15c106cd373095bb) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0071b6100, 0x33, 0x0, 0x0, 0xc002763ce0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7595 [IO wait]: internal/poll.runtime_pollWait(0x7f6d0c1ce328, 0x72, 0xffffffffffffffff) /usr/local/go/src/runtime/netpoll.go:182 +0x56 internal/poll.(*pollDesc).wait(0xc003d05d98, 0x72, 0x700, 0x75b, 0xffffffffffffffff) /usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x9b internal/poll.(*pollDesc).waitRead(...) /usr/local/go/src/internal/poll/fd_poll_runtime.go:92 internal/poll.(*FD).Read(0xc003d05d80, 0xc003592800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/internal/poll/fd_unix.go:169 +0x19b net.(*netFD).Read(0xc003d05d80, 0xc003592800, 0x75b, 0x75b, 0xc00182d938, 0x701, 0xc004371680) /usr/local/go/src/net/fd_unix.go:202 +0x4f net.(*conn).Read(0xc0032be5a8, 0xc003592800, 0x75b, 0x75b, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:177 +0x69 github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux.(*MuxConn).Read(0xc00325be60, 0xc003592800, 0x75b, 0x75b, 0xc00359280d, 0x5b, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/cockroachdb/cmux/cmux.go:218 +0x111 crypto/tls.(*atLeastReader).Read(0xc0077a26c0, 0xc003592800, 0x75b, 0x75b, 0x78, 0x482c600, 0xc00182da50) /usr/local/go/src/crypto/tls/conn.go:761 +0x60 bytes.(*Buffer).ReadFrom(0xc004371758, 0x482c340, 0xc0077a26c0, 0x6fcab5, 0x3aee1e0, 0x3d29400) /usr/local/go/src/bytes/buffer.go:207 +0xbd crypto/tls.(*Conn).readFromUntil(0xc004371500, 0x7f6d0700c008, 0xc00325be60, 0x5, 0xc00325be60, 0x5b) /usr/local/go/src/crypto/tls/conn.go:783 +0xf8 crypto/tls.(*Conn).readRecordOrCCS(0xc004371500, 0x40e5100, 0xc004371638, 0x3) /usr/local/go/src/crypto/tls/conn.go:590 +0x125 crypto/tls.(*Conn).readRecord(...) /usr/local/go/src/crypto/tls/conn.go:558 crypto/tls.(*Conn).Read(0xc004371500, 0xc004a96000, 0x8000, 0x8000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:1236 +0x137 bufio.(*Reader).Read(0xc001b07260, 0xc004d27a78, 0x9, 0x9, 0xc003a760c8, 0xc004501200, 0x52) /usr/local/go/src/bufio/bufio.go:223 +0x23e io.ReadAtLeast(0x482c140, 0xc001b07260, 0xc004d27a78, 0x9, 0x9, 0x9, 0x59980b3493438c01, 0x5900000000000000, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.readFrameHeader(0xc004d27a78, 0x9, 0x9, 0x482c140, 0xc001b07260, 0x0, 0x300000000, 0xc000c26000, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:237 +0x88 github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc004d27a40, 0xc001b14150, 0xc001b14150, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/golang.org/x/net/http2/frame.go:492 +0xa1 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams(0xc001ece180, 0xc000c59ce0, 0x40e4358) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/http2_server.go:431 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams(0xc00193c600, 0x48fef60, 0xc001ece180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:713 +0xdd github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn.func1(0xc00193c600, 0x48fef60, 0xc001ece180) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:675 +0x43 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleRawConn /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:674 +0x580 goroutine 236385 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc0061dd440, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc0033bbe60, 0xc000ee6738, 0xc0032fc3e0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc004e72480, 0x33, 0x0, 0x0, 0xc003a22090) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6317 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).read(0xc003c090e0, 0xc003c974f0, 0x5, 0x5, 0x0, 0x0, 0xc00314e400) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:146 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc003c090e0, 0xc003c974f0, 0x5, 0x5, 0xc11b10, 0xc0003571e0, 0xc0043b1af8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:140 +0x1a6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc003bd7cb0, 0xc003c974f0, 0x5, 0x5, 0x3b, 0xc0043b1b20, 0xc3149c) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc003bd7cb0, 0xc003c974f0, 0x5, 0x5, 0x5, 0xc0043b1bd0, 0xb6db6db6db6d, 0x39) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc0037f7a00, 0xc003c974f0, 0x5, 0x5, 0x203000, 0x203000, 0x72b5601) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc003c974e0, 0x7fffffff, 0x732a2f, 0xc000bdb980, 0x4, 0xc0043b1c70, 0x733ecb, 0xc00146b7a0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc003c974e0, 0xc0037f7a00, 0x0, 0x0, 0x7fffffff, 0xc0043b1e30, 0x4886820, 0x72b56a8, 0x8, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc003c974e0, 0x7f6d0c1990d0, 0xc000292db0, 0xc0037f7a00, 0x0, 0x0, 0x3de86e0, 0xc00380a3c0, 0x7fffffff, 0xc0043b1e30, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*serverStream).RecvMsg(0xc00241cb40, 0x3de86e0, 0xc00380a3c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:1458 +0x14e github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchServer).Recv(0xc000d57880, 0x489e1a0, 0xc003bd7d70, 0xc00314e400) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:710 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*lockedRaftMessageResponseStream).Recv(...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1.1(0x48fa580, 0xc000d57880, 0xc001460780, 0x489e1a0, 0xc003bd7d70, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:342 +0x7f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch.func1.1(0x489e1a0, 0xc003bd7d70) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:378 +0x5d github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc000d578a0, 0xc000937c20, 0xc003bd7d40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 6316 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftMessageBatch(0xc001460780, 0x48fa580, 0xc000d57880, 0x485c120, 0xc001460780) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:384 +0x1b8 github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftMessageBatch_Handler(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00241cb40, 0x0, 0xc0026126c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:691 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00241cb40, 0xc003c97540, 0x40e1ba8, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc001460780, 0x48eb940, 0xc00241cb40, 0xc003c97540, 0x40e1ba8, 0x489e1a0, 0xc003bd7d10) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0037f7a00, 0xc00111b2c0, 0x6fa49e0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0037f7a00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004492f0, 0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0037f7a00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 137494 [select]: github.com/cockroachdb/cockroach/pkg/sql/stats.(*Refresher).Start.func1.1(0x489e1a0, 0xc001582e10) /go/src/github.com/cockroachdb/cockroach/pkg/sql/stats/automatic_stats.go:258 +0x11f github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc0019e4510, 0x489e1a0, 0xc001582e10, 0xc00190f1d0, 0x2a, 0x0, 0x0, 0xc0046fbe00) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 235871 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc004a32180, 0x0, 0x1, 0xc00616ed80, 0xc000cabe28) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc0, 0xc00073fe60, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc007087840, 0x33, 0x0, 0x0, 0xc005c045a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 234898 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc002e2daa0, 0x0, 0x1, 0xc001b67ec0, 0xbab23ccf) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc0, 0xc004c58b40, 0xc003f20800, 0x15c106cd26194fa7) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc005658280, 0x33, 0x0, 0x0, 0xc006f708a0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 6811 [select]: github.com/cockroachdb/cockroach/pkg/rpc.(*Context).runHeartbeat(0xc001dc04e0, 0xc0014c38c0, 0xc000448e90, 0xf, 0xc000885aa0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:933 +0x276 github.com/cockroachdb/cockroach/pkg/rpc.(*Context).grpcDialNodeInternal.func1.1.1(0x489e1a0, 0xc001d55110) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:878 +0x79 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001dbc6c0, 0xc0019e4510, 0xc002648c80) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 235655 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc3, 0x490bae0, 0xc006844d80, 0x0, 0x1, 0x7, 0xc0061ab890) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc3, 0xc00525cc60, 0xc00658ea20, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc001c6c480, 0x33, 0x0, 0x0, 0xc004fe9590) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 236265 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6a, 0x490bae0, 0xc004a32660, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6a, 0xc0058c87e0, 0xc004ce8800, 0x15c106cd34be85bd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc007087900, 0x33, 0x0, 0x0, 0xc005c05b90) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 238944 [semacquire]: sync.runtime_SemacquireMutex(0x726cb6c, 0xc93162c00) /usr/local/go/src/runtime/sema.go:71 +0x3d sync.(*Mutex).Lock(0x726cb68) /usr/local/go/src/sync/mutex.go:134 +0x109 github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x726cb20, 0xc000000001, 0x6bde2b1, 0x1a, 0x596, 0xc007a003c0, 0xa0) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:857 +0x206 github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x489e1a0, 0xc0060ff050, 0xc000000001, 0x2, 0x4020db4, 0x3a, 0xc0010a3aa8, 0x3, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x489e1a0, 0xc0060ff050, 0x1, 0xc000000001, 0x4020db4, 0x3a, 0xc0010a3aa8, 0x3, 0x3) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c github.com/cockroachdb/cockroach/pkg/util/log.Infof(...) /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:95 github.com/cockroachdb/cockroach/pkg/storage.execChangeReplicasTxn.func1(0x489e1a0, 0xc0060ff050, 0xc0065f46c0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1430 +0x297 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn.func1(0x489e1a0, 0xc0060ff050, 0xc0065f46c0, 0x0, 0x15c106cd54760676) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:681 +0x43 github.com/cockroachdb/cockroach/pkg/internal/client.(*Txn).exec(0xc0065f46c0, 0x489e1a0, 0xc0060ff050, 0xc007c9f680, 0x1, 0xc0065f46c0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/txn.go:704 +0xd9 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Txn(0xc001081880, 0x489e1a0, 0xc0060ff050, 0xc0065f45a0, 0x4, 0x4) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:680 +0xe4 github.com/cockroachdb/cockroach/pkg/storage.execChangeReplicasTxn(0x489e1a0, 0xc0060ff050, 0xc000932000, 0xc002f3a058, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005071ba4, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1423 +0x64f github.com/cockroachdb/cockroach/pkg/storage.(*Replica).atomicReplicationChange(0xc0035be800, 0x489e1a0, 0xc0060ff050, 0xc002f3a058, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005071bb0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1165 +0x5da github.com/cockroachdb/cockroach/pkg/storage.(*Replica).changeReplicasImpl(0xc0035be800, 0x489e1a0, 0xc0060ff050, 0xc002f3a058, 0xc000000002, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005071bb0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:966 +0x4eb github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ChangeReplicas(0xc0035be800, 0x489e1a0, 0xc0060ff050, 0xc002f3a058, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc005071bb0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:902 +0x224 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc0035be800, 0x489e1a0, 0xc0060ff050, 0xc0072aaf00, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1324 +0x3c8 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc0035be800, 0x489e1a0, 0xc0060ff020, 0xc1, 0x15c106cd547584db, 0x0, 0x200000002, 0x3, 0xc005071b2c, 0xc1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc0035be800, 0x489e1a0, 0xc0060ff020, 0x15c106cd547584db, 0x0, 0x200000002, 0x3, 0xc005071b2c, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000932000, 0x489e1a0, 0xc0060feea0, 0x15c106cd547584db, 0x0, 0x200000002, 0x3, 0xc005071b2c, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc000f63cc0, 0x489e1a0, 0xc0060feea0, 0x0, 0x0, 0x200000002, 0x3, 0xc005071b2c, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc0060fede0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000937c20, 0x489e1a0, 0xc0060fede0, 0x3f956b7, 0x10, 0xc0035db710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc00174a000, 0x489e1a0, 0xc0060fede0, 0xc0072aae80, 0xc0060fede0, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc00174a000, 0x489e1a0, 0xc0060fed80, 0xc0072aae80, 0xc00174a000, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc0060fed80, 0x3f4fce0, 0xc0072aae80, 0x3f4fce0, 0xc0072aae80, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc0060fed80, 0x3f4fce0, 0xc0072aae80, 0xc00518efa0, 0xc00518efc0, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc0060fed80, 0x3f4fce0, 0xc0072aae80, 0xc00518efa0, 0xc00518efc0, 0x6ff238, 0x20, 0x3b5a920, 0xc003276101) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc0060fed80, 0x3f4fce0, 0xc0072aae80, 0xc00518efa0, 0xc00518efc0, 0xc002792bc8, 0x7edeba, 0x3d2f600, 0xc0060fed80) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc00174a000, 0x489e1a0, 0xc0060fed80, 0xc0032761e0, 0xc000d57350, 0x489e1a0, 0xc0060fed80, 0xc00241a000, 0x5b) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0073ecb00, 0xc00111ba10, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0073ecb00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0004492f0, 0xc00193c600, 0x48fef60, 0xc00260a480, 0xc0073ecb00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 236343 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x30, 0x490bae0, 0xc000c253e0, 0x0, 0x1, 0x38, 0x50) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x30, 0xc006cf8480, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00478f740, 0x33, 0x0, 0x0, 0xc003e03170) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 236179 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x30, 0x490bae0, 0xc0026af8c0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x30, 0xc004fd3b00, 0xc001858f38, 0xc00685f0a0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc003edec00, 0x33, 0x0, 0x0, 0xc004d4e7b0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 234214 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6b, 0x490bae0, 0xc002cb1260, 0x0, 0x1, 0x8, 0xc005b8bf70) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6b, 0xc003324000, 0xc00353f200, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc002a4d0c0, 0x33, 0x0, 0x0, 0xc001d54f60) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 234213 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc002cb1140, 0x0, 0x1, 0x8, 0xc005b8bf70) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc0039fcea0, 0xc00353f200, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc002a4d080, 0x33, 0x0, 0x0, 0xc001d54b40) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 237948 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc2, 0x490bae0, 0xc007c2a360, 0x0, 0x1, 0x38, 0x50) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc2, 0xc0046f4fc0, 0x482c600, 0xc0066fe65c) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0063affc0, 0x33, 0x0, 0x0, 0xc006457da0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 236140 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc006437200, 0x0, 0x1, 0xc005804c00, 0x53) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc0019d26c0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00655c7c0, 0x33, 0x0, 0x0, 0xc0047b5c50) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 239670 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc0027e6190, 0xc005f793d0, 0x5, 0x5, 0x0, 0xc006f9f998, 0xde3d07) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc0027e6190, 0xc005f793d0, 0x5, 0x5, 0xc11b10, 0xc0070cc6c0, 0xc006f9f5c8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc00628c9f0, 0xc005f793d0, 0x5, 0x5, 0xd25cf03, 0xc006f9f5f0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc00628c9f0, 0xc005f793d0, 0x5, 0x5, 0x5, 0xc006928069, 0x4, 0x43) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc006946b00, 0xc005f793d0, 0x5, 0x5, 0x1, 0x0, 0xc006f9f6e8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc005f793c0, 0x7fffffff, 0x72b56a8, 0x3dec160, 0xc000364d20, 0xc006f9f748, 0x7ee108, 0xc004cf0db0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc005f793c0, 0xc006946b00, 0x0, 0x0, 0x7fffffff, 0xc006f9f908, 0x4886820, 0x72b56a8, 0x3dec160, 0xc000364d20, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc005f793c0, 0x7f6d0c1990d0, 0xc000292db0, 0xc006946b00, 0x0, 0x0, 0x3de8b60, 0xc007322320, 0x7fffffff, 0xc006f9f908, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc006991280, 0x3de8b60, 0xc007322320, 0xc006f9f908, 0x1, 0xc007322320) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc006991280, 0x40e, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc005284b40, 0xc006f9fa50, 0xc006f9fa40, 0x83, 0x10) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc005284b40, 0x3de8b60, 0xc007322320, 0xc006f9ff10, 0x4) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftSnapshotClient).Recv(0xc008422d20, 0xc004cf0ff0, 0x1, 0xc000000001) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:674 +0x62 github.com/cockroachdb/cockroach/pkg/storage.sendSnapshot(0x489e1a0, 0xc004cf0ff0, 0xc000a15538, 0xc000a32000, 0x7f6d05d41028, 0xc008422d20, 0x482f3e0, 0xc000c183c0, 0x13, 0x6, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:1042 +0xe84 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).SendSnapshot(0xc0002a8960, 0x489e1a0, 0xc004cf0ff0, 0xc000a15538, 0xc000c183c0, 0x13, 0x6, 0xc0028e5ce0, 0xc0027e6050, 0xc005a28930, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:681 +0x27b github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendSnapshot(0xc00511b800, 0x489e1a0, 0xc004cf0ff0, 0x300000003, 0xc000000003, 0xc006d1804c, 0x200000001, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1703 +0x5e1 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).atomicReplicationChange(0xc00511b800, 0x489e1a0, 0xc004cf0ff0, 0xc005967420, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc006d18010, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1151 +0x1f9 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).changeReplicasImpl(0xc00511b800, 0x489e1a0, 0xc004cf0ff0, 0xc003166058, 0xc000000002, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc006d18010, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:966 +0x4eb github.com/cockroachdb/cockroach/pkg/storage.(*Replica).ChangeReplicas(0xc00511b800, 0x489e1a0, 0xc004cf0ff0, 0xc003166058, 0x2, 0x3f8dc67, 0xd, 0x0, 0x0, 0xc006d18010, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:902 +0x224 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc00511b800, 0x489e1a0, 0xc004cf0ff0, 0xc0068a8600, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1324 +0x3c8 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc00511b800, 0x489e1a0, 0xc004cf0fc0, 0xc3, 0x15c106cd51b14940, 0x0, 0x100000001, 0x1, 0xc0003c6de0, 0xc3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc00511b800, 0x489e1a0, 0xc004cf0fc0, 0x15c106cd51b14940, 0x0, 0x100000001, 0x1, 0xc0003c6de0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a15500, 0x489e1a0, 0xc004cf0f30, 0x15c106cd51b14940, 0x0, 0x100000001, 0x1, 0xc0003c6de0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc0003b5f40, 0x489e1a0, 0xc004cf0f30, 0x0, 0x0, 0x100000001, 0x1, 0xc0003c6de0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc004cf0f30, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000a30090, 0x489e1a0, 0xc004cf0f30, 0x3f956b7, 0x10, 0xc00308ffa0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000786500, 0x489e1a0, 0xc004cf0f30, 0xc0068a8580, 0xc004cf0f30, 0xc004cf0cf0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000786500, 0x489e1a0, 0xc004cf0f00, 0xc0068a8580, 0xc004cf0de0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/rpc.internalClientAdapter.Batch(...) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:540 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc006f09980, 0x489e1a0, 0xc004cf0f00, 0x1, 0x485fe20, 0xc0003623a0, 0x0, 0x0, 0x100000001, 0x1, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc006f09980, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x100000001, 0x1, 0xc0003c6de0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc000c42000, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0001560e0, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc000156080, 0x489e1a0, 0xc004cf0de0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e1a0, 0xc004cf0de0, 0xc003091ba0, 0xc0021a3180, 0xc003166000, 0xc007530258) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc000156080, 0x489e1a0, 0xc004cf0de0, 0xc0021a3180, 0xc007530258, 0x2) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminChangeReplicas(0xc000156080, 0x489e1a0, 0xc004cf0de0, 0x3d0b580, 0xc005ac1b40, 0xc3, 0xc007530258, 0x2, 0x8, 0xc007530260, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:582 +0x268 github.com/cockroachdb/cockroach/pkg/storage.(*Store).AdminRelocateRange(0xc000a15500, 0x489e1a0, 0xc004cf0de0, 0xc3, 0xc007530258, 0x2, 0x8, 0xc007530260, 0x3, 0x8, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_command.go:1984 +0x1628 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).executeAdminBatch(0xc00511b800, 0x489e1a0, 0xc004cf0de0, 0xc0068a8400, 0xa, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:1335 +0x803 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).sendWithRangeID(0xc00511b800, 0x489e1a0, 0xc004cf0db0, 0xc3, 0x15c106cd51b008cb, 0x0, 0x100000001, 0x1, 0xc00601eb40, 0xc3, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:581 +0x6d6 github.com/cockroachdb/cockroach/pkg/storage.(*Replica).Send(0xc00511b800, 0x489e1a0, 0xc004cf0db0, 0x15c106cd51b008cb, 0x0, 0x100000001, 0x1, 0xc00601eb40, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica.go:523 +0x83 github.com/cockroachdb/cockroach/pkg/storage.(*Store).Send(0xc000a15500, 0x489e1a0, 0xc004cf0d50, 0x15c106cd51b008cb, 0x0, 0x100000001, 0x1, 0xc00601eb40, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/store.go:3010 +0x5c7 github.com/cockroachdb/cockroach/pkg/storage.(*Stores).Send(0xc0003b5f40, 0x489e1a0, 0xc004cf0d50, 0x0, 0x0, 0x100000001, 0x1, 0xc00601eb40, 0xc3, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/storage/stores.go:181 +0xce github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal.func1(0x489e1a0, 0xc004cf0d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:916 +0x1e8 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc000a30090, 0x489e1a0, 0xc004cf0d20, 0x3f956b7, 0x10, 0xc003093710, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb github.com/cockroachdb/cockroach/pkg/server.(*Node).batchInternal(0xc000786500, 0x489e1a0, 0xc004cf0d20, 0xc0068a8380, 0xc004cf0d20, 0xc00004e000, 0x3afd920) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:904 +0x195 github.com/cockroachdb/cockroach/pkg/server.(*Node).Batch(0xc000786500, 0x489e1a0, 0xc004cf0cf0, 0xc0068a8380, 0xc000786500, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/server/node.go:942 +0x9f github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler.func1(0x489e1a0, 0xc004cf0cf0, 0x3f4fce0, 0xc0068a8380, 0x3f4fce0, 0xc0068a8380, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9981 +0x86 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingServerInterceptor.func1(0x489e1a0, 0xc004cf0cf0, 0x3f4fce0, 0xc0068a8380, 0xc005ac1940, 0xc005ac1960, 0x0, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/server.go:44 +0x9df github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func2(0x489e1a0, 0xc004cf0cf0, 0x3f4fce0, 0xc0068a8380, 0xc005ac1940, 0xc005ac1960, 0x6ff238, 0x20, 0x3b5a920, 0xc0051e7901) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:221 +0xb6 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func4(0x489e1a0, 0xc004cf0cf0, 0x3f4fce0, 0xc0068a8380, 0xc005ac1940, 0xc005ac1960, 0xc00182fbc8, 0x7edeba, 0x3d2f600, 0xc004cf0cf0) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:251 +0xac github.com/cockroachdb/cockroach/pkg/roachpb._Internal_Batch_Handler(0x3f156e0, 0xc000786500, 0x489e1a0, 0xc004cf0cf0, 0xc0051e7980, 0xc00037c170, 0x489e1a0, 0xc004cf0cf0, 0xc001d12c00, 0x2a) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9983 +0x158 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processUnaryRPC(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc004d98300, 0xc000c30630, 0x6f96c80, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:999 +0x466 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc004d98300, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1279 +0xda6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0002c1090, 0xc000c12a80, 0x48fef60, 0xc001c10900, 0xc004d98300) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 238864 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot(0xc0018a82d0, 0x48fa640, 0xc008422e70, 0x485c120, 0xc0018a82d0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:420 +0x1cf github.com/cockroachdb/cockroach/pkg/storage._MultiRaft_RaftSnapshot_Handler(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc004d74f00, 0x0, 0xc002e646c0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:717 +0xad github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func3(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc004d74f00, 0xc005f796a0, 0x40e1bb0, 0x3c89ac0, 0xc0001bc301) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:238 +0x100 github.com/cockroachdb/cockroach/pkg/rpc.NewServerWithInterceptor.func5(0x3ee0fa0, 0xc0018a82d0, 0x48eb940, 0xc004d74f00, 0xc005f796a0, 0x40e1bb0, 0x489e1a0, 0xc00628cd50) /go/src/github.com/cockroachdb/cockroach/pkg/rpc/context.go:263 +0xb6 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).processStreamingRPC(0xc00262b200, 0x48fef60, 0xc003110000, 0xc006946c00, 0xc0009a98f0, 0x6fa4a00, 0x0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1210 +0x462 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).handleStream(0xc00262b200, 0x48fef60, 0xc003110000, 0xc006946c00, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:1283 +0xd3f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc0003abbd0, 0xc00262b200, 0x48fef60, 0xc003110000, 0xc006946c00) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:717 +0x9f created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*Server).serveStreams.func1 /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/server.go:715 +0xa1 goroutine 238557 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).waitOnHeader(0xc006c3e100, 0xc006d19f58, 0xc00588f298) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:267 +0xcc github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).RecvCompress(...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:278 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc0075bd600, 0x3e99440, 0xc00781e5a0, 0xc00793b2f0, 0x4886820, 0x72b56a8) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:854 +0x70c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc0075bd600, 0xc006c977c0, 0x37) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc0060bf680, 0xc00793b438, 0xc00793b428, 0xc0047051d0, 0xc006d19f58) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:577 +0x7c github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc0060bf680, 0x3e99440, 0xc00781e5a0, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.invoke(0x489e160, 0xc007285680, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0075bd580, 0x3e99440, 0xc00781e5a0, 0xc002256000, 0xc00172a4c0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:73 +0x141 github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc.OpenTracingClientInterceptor.func1(0x489e160, 0xc007285680, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0075bd580, 0x3e99440, 0xc00781e5a0, 0xc002256000, 0x40e43f0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/github.com/grpc-ecosystem/grpc-opentracing/go/otgrpc/client.go:47 +0xacf github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ClientConn).Invoke(0xc002256000, 0x489e160, 0xc007285680, 0x3fc5646, 0x21, 0x3f4fce0, 0xc0075bd580, 0x3e99440, 0xc00781e5a0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/call.go:35 +0x10a github.com/cockroachdb/cockroach/pkg/roachpb.(*internalClient).Batch(0xc003316ed0, 0x489e160, 0xc007285680, 0xc0075bd580, 0x0, 0x0, 0x0, 0x3, 0x489e160, 0xc007285680) /go/src/github.com/cockroachdb/cockroach/pkg/roachpb/api.pb.go:9919 +0xd2 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).sendBatch(0xc00694e200, 0x489e160, 0xc007285680, 0x3, 0x485a560, 0xc003316ed0, 0x0, 0x0, 0x300000003, 0x2, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:199 +0x128 github.com/cockroachdb/cockroach/pkg/kv.(*grpcTransport).SendNext(0xc00694e200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x300000003, 0x2, 0xc0070113d4, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/transport.go:168 +0x15d github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendToReplicas(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1670 +0x2a3 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendRPC(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc1, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:436 +0x196 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendSingleRange(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:518 +0x2f2 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).sendPartialBatch(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1441 +0x3ce github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).divideAndSendBatchToRanges(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:1090 +0x1536 github.com/cockroachdb/cockroach/pkg/kv.(*DistSender).Send(0xc0015bd200, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/kv/dist_sender.go:732 +0xd39 github.com/cockroachdb/cockroach/pkg/internal/client.(*CrossRangeTxnWrapperSender).Send(0xc0010818e0, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:218 +0xaf github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).sendUsingSender(0xc001081880, 0x489e160, 0xc007285680, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:718 +0x122 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).send(...) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:700 github.com/cockroachdb/cockroach/pkg/internal/client.sendAndFill(0x489e160, 0xc007285680, 0xc0058913a8, 0xc00416eb00, 0x8, 0xc006d19e90) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:643 +0xf7 github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).Run(0xc001081880, 0x489e160, 0xc007285680, 0xc00416eb00, 0x1, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:666 +0x9c github.com/cockroachdb/cockroach/pkg/internal/client.(*DB).AdminRelocateRange(0xc001081880, 0x489e160, 0xc007285680, 0x3cd5280, 0xc0050db640, 0xc006d19ef8, 0x1, 0x1, 0xc00624f8e0, 0x1) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/db.go:605 +0xa5 github.com/cockroachdb/cockroach/pkg/storage.(*mergeQueue).process(0xc0003b2220, 0x489e160, 0xc007285680, 0xc006363800, 0xc0065f7720, 0x3f8e738, 0xd) /go/src/github.com/cockroachdb/cockroach/pkg/storage/merge_queue.go:341 +0xda3 github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica.func1(0x489e160, 0xc007285680, 0xdf8475800, 0x489e160) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:890 +0x2ad github.com/cockroachdb/cockroach/pkg/util/contextutil.RunWithTimeout(0x489e160, 0xc007285680, 0xc001ce50c0, 0x1f, 0xdf8475800, 0xc003bd6d20, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/util/contextutil/context.go:135 +0x9d github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processReplica(0xc001600000, 0x489e1a0, 0xc003bd6cf0, 0x4906de0, 0xc006363800, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:852 +0x26a github.com/cockroachdb/cockroach/pkg/storage.(*baseQueue).processLoop.func1.2(0x489e1a0, 0xc003bd6c90) /go/src/github.com/cockroachdb/cockroach/pkg/storage/queue.go:779 +0xd9 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000937c20, 0x489e1a0, 0xc003bd6c90, 0xc00192af00, 0x29, 0x0, 0x0, 0xc003bd6cc0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 7697 [select]: github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue(0xc0018a82d0, 0x2, 0xc001c98060, 0xc0046b6040, 0x48fa520, 0xc002aa20e0, 0x48fa501, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:485 +0x44f github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).startProcessNewQueue.func1(0x489e1a0, 0xc001582fc0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:619 +0x216 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc002aa2070, 0xc0019e4510, 0xc001582d20) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 9966 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc002d9df40, 0xc0025c4710, 0x5, 0x5, 0xa457a1, 0x3d10fa0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:158 +0xe5 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc002d9df40, 0xc0025c4710, 0x5, 0x5, 0xc11b10, 0xc00255eb60, 0xc0053fd978) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:138 +0x127 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001799f80, 0xc0025c4710, 0x5, 0x5, 0x74cff0, 0xc0053fd9a0, 0xc301cc) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:435 +0x55 io.ReadAtLeast(0x48313e0, 0xc001799f80, 0xc0025c4710, 0x5, 0x5, 0x5, 0xf, 0xc00107e1e0, 0x0) /usr/local/go/src/io/io.go:310 +0x88 io.ReadFull(...) /usr/local/go/src/io/io.go:329 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport.(*Stream).Read(0xc003a70000, 0xc0025c4710, 0x5, 0x5, 0x83e500, 0xc0076048c0, 0xc0053fda78) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/internal/transport/transport.go:419 +0xc8 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*parser).recvMsg(0xc0025c4700, 0x7fffffff, 0x3c06120, 0xc00107e1e0, 0xc0053fda0c, 0x2, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:508 +0x63 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recvAndDecompress(0xc0025c4700, 0xc003a70000, 0x0, 0x0, 0x7fffffff, 0xc0053fdcb8, 0x4886820, 0x72b56a8, 0x0, 0xc0045b8fa0, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:639 +0x4d github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.recv(0xc0025c4700, 0x7f6d0c1990d0, 0xc000292db0, 0xc003a70000, 0x0, 0x0, 0x3de8800, 0xc006f08c40, 0x7fffffff, 0xc0053fdcb8, ...) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/rpc_util.go:684 +0x9b github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*csAttempt).recvMsg(0xc002982700, 0x3de8800, 0xc006f08c40, 0xc0053fdcb8, 0xc0056a57c0, 0xc006f08c40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:868 +0xed github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc002982700, 0xc000080570, 0x203001) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:719 +0x46 github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).withRetry(0xc004020b40, 0xc0053fde00, 0xc0053fddf0, 0x40e2178, 0x40) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:573 +0x29f github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*clientStream).RecvMsg(0xc004020b40, 0x3de8800, 0xc006f08c40, 0x2c5e8e0, 0xc000932000) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/stream.go:718 +0x100 github.com/cockroachdb/cockroach/pkg/storage.(*multiRaftRaftMessageBatchClient).Recv(0xc001c0e600, 0x489e1a0, 0xc000930210, 0xc004c60d80) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft.pb.go:643 +0x62 github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1.1(0x48fa520, 0xc001c0e600, 0xc00407cbc0, 0xc001460780, 0x489e1a0, 0xc000930210, 0x71f47f, 0xc000000008) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:459 +0x3b github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).processQueue.func1.1(0x489e1a0, 0xc000930210) /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:474 +0x66 github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker.func1(0xc001c0e620, 0xc000937c20, 0xc001799fb0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:196 +0xfb created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunWorker /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:189 +0xa8 goroutine 10250 [select]: github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0037c0600) /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:115 +0x110 created by github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc.newCCBalancerWrapper /go/src/github.com/cockroachdb/cockroach/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:106 +0x16a goroutine 235869 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xc0, 0x490bae0, 0xc00362e1e0, 0x0, 0x1, 0x0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xc0, 0xc00298afc0, 0x3bfc3a0, 0x0) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0070871c0, 0x33, 0x0, 0x0, 0xc00453a8d0) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 234259 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6e, 0x490bae0, 0xc002e36d80, 0x0, 0x1, 0x0, 0xc004d0d270) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6e, 0xc00330f9e0, 0xc003d58800, 0x15c106cd233b9309) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc00682f680, 0x33, 0x0, 0x0, 0xc001826930) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 235506 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0xbf, 0x490bae0, 0xc0073f79e0, 0x0, 0x1, 0x0, 0x3de2740) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0xbf, 0xc0056c3e60, 0xc0077a5800, 0x15c106cd28c5f30b) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc0070861c0, 0x33, 0x0, 0x0, 0xc005f73800) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 goroutine 240092 [select]: github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher.(*RequestBatcher).Send(0xc00029a380, 0x489e120, 0xc00010c010, 0x6, 0x490bae0, 0xc007861bc0, 0x0, 0x1, 0x7, 0xc002238a20) /go/src/github.com/cockroachdb/cockroach/pkg/internal/client/requestbatcher/batcher.go:240 +0x1e9 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).gcTxnRecord(0xc000be9a00, 0x489e120, 0xc00010c010, 0x6, 0xc002425200, 0xc002a1d000, 0x15c106cd56739c0d) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:772 +0x372 github.com/cockroachdb/cockroach/pkg/storage/intentresolver.(*IntentResolver).cleanupFinishedTxnIntents.func2(0x489e120, 0xc00010c010) /go/src/github.com/cockroachdb/cockroach/pkg/storage/intentresolver/intent_resolver.go:812 +0x6b github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc000a30090, 0x489e120, 0xc00010c010, 0xc008409840, 0x33, 0x0, 0x0, 0xc005b3f620) /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6 created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131 **************************************************************************** This node experienced a fatal error (printed above), and as a result the process is terminating. Fatal errors can occur due to faulty hardware (disks, memory, clocks) or a problem in CockroachDB. With your help, the support team at Cockroach Labs will try to determine the root cause, recommend next steps, and we can improve CockroachDB based on your report. Please submit a crash report by following the instructions here: https://github.com/cockroachdb/cockroach/issues/new/choose If you would rather not post publicly, please contact us directly at: support@cockroachlabs.com The Cockroach Labs team appreciates your feedback. W190903 20:13:44.288017 239649 storage/replica_command.go:1917 [n2,s2,r193/3:/Table/95/1/{5-6}] while transferring lease: unable to find store 1 in range r193:/Table/95/1/{5-6} [(n3,s3):2, (n2,s2):3, next=4, gen=167] I190903 20:13:44.352638 239649 storage/replica_command.go:1430 [n2,s2,r193/3:/Table/95/1/{5-6}] change replicas (add [] remove [(n3,s3):2]): existing descriptor r193:/Table/95/1/{5-6} [(n3,s3):2, (n2,s2):3, next=4, gen=167] ERROR: exit status 255 context canceled Error: exit status 1 1186 runs completed, 0 failures, over 23m46s context canceled FAIL Makefile:973: recipe for target 'roachprod-stress' failed make: *** [roachprod-stress] Error 1