GC worker blocks forever

[youjiali1995]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do?

Test green gc with 1 TiDB and 4 TiKV.

2. What did you expect to see?

3. What did you see instead?

GC worker blocks forever.
1 goroutine blocks here:

 0  0x0000000000437430 in runtime.gopark
    at /usr/lib/go-1.14.1/src/runtime/proc.go:305
 1  0x00000000004052fd in runtime.chansend
    at /usr/lib/go-1.14.1/src/runtime/chan.go:236
 2  0x00000000004050c5 in runtime.chansend1
    at /usr/lib/go-1.14.1/src/runtime/chan.go:127
 3  0x0000000002469201 in github.com/pingcap/tidb/store/tikv/gcworker.(*GCWorker).physicalScanAndResolveLocks
    at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:1276
 4  0x00000000024652f0 in github.com/pingcap/tidb/store/tikv/gcworker.(*GCWorker).resolveLocksPhysical
    at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:1055
 5  0x00000000024624aa in github.com/pingcap/tidb/store/tikv/gcworker.(*GCWorker).resolveLocks
    at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:893
 6  0x000000000246f2e9 in github.com/pingcap/tidb/store/tikv/gcworker.RunDistributedGCJob
    at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:1702
 7  0x0000000002474160 in github.com/pingcap/tipocket/tests/resolve-lock.(*resolveLockClient).Start
    at ./tests/resolve-lock/resolve_lock.go:224
 8  0x000000000186f08a in github.com/pingcap/tipocket/pkg/control.(*Controller).RunSelfScheduled.func2
    at ./pkg/control/control.go:262
 9  0x0000000000cbd829 in golang.org/x/sync/errgroup.(*Group).Go.func1
    at /home/youjiali1995/go/pkg/mod/golang.org/x/sync@v0.0.0-20190911185100-cd5d95a43a6e/errgroup/errgroup.go:57
10  0x0000000000467091 in runtime.goexit
    at /usr/lib/go-1.14.1/src/runtime/asm_amd64.s:1373

And 4 goroutines block here:

0  0x0000000000437430 in runtime.gopark
   at /usr/lib/go-1.14.1/src/runtime/proc.go:305
1  0x00000000004052fd in runtime.chansend
   at /usr/lib/go-1.14.1/src/runtime/chan.go:236
2  0x00000000004050c5 in runtime.chansend1
   at /usr/lib/go-1.14.1/src/runtime/chan.go:127
3  0x0000000002470436 in github.com/pingcap/tidb/store/tikv/gcworker.(*mergeLockScanner).physicalScanLocksForStore
   at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:1950
4  0x0000000002471b7b in github.com/pingcap/tidb/store/tikv/gcworker.(*mergeLockScanner).Start.func1
   at /home/youjiali1995/go/pkg/mod/github.com/youjiali1995/tidb@v1.1.0-beta.0.20200411103223-b7c621421015/store/tikv/gcworker/gc_worker.go:1855
5  0x0000000000467091 in runtime.goexit
   at /usr/lib/go-1.14.1/src/runtime/asm_amd64.s:1373

4. What version of TiDB are you using? (tidb-server -V or run select tidb_version(); on TiDB)

4.0