Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

IT goroutine leak detected in integration ddl #30332

Closed
aytrack opened this issue Dec 2, 2021 · 3 comments · Fixed by #30369 or #30467
Closed

IT goroutine leak detected in integration ddl #30332

aytrack opened this issue Dec 2, 2021 · 3 comments · Fixed by #30369 or #30467
Assignees
@lcwangchao
Labels
component/test severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@aytrack
Copy link
Contributor

aytrack commented Dec 2, 2021

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

in ci https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_integration_ddl_test/detail/tidb_ghpr_integration_ddl_test/6756/pipeline

2021-12-01T09:40:42.083Z] goleak: Errors on successful test run: found unexpected goroutines:

[2021-12-01T09:40:42.083Z] [Goroutine 529 in state select, with context.propagateCancel.func1 on top of the stack:

[2021-12-01T09:40:42.083Z] goroutine 529 [select]:

[2021-12-01T09:40:42.083Z] context.propagateCancel.func1(0x3f6ae58, 0xc000698580, 0x3f43140, 0xc00057e400)

[2021-12-01T09:40:42.083Z] 	/usr/local/go/src/context/context.go:279 +0xab

[2021-12-01T09:40:42.083Z] created by context.propagateCancel

[2021-12-01T09:40:42.083Z] 	/usr/local/go/src/context/context.go:278 +0x1fb

[2021-12-01T09:40:42.083Z] 

[2021-12-01T09:40:42.083Z]  Goroutine 72 in state select, with go.opencensus.io/stats/view.(*worker).start on top of the stack:

[2021-12-01T09:40:42.083Z] goroutine 72 [select]:

[2021-12-01T09:40:42.083Z] go.opencensus.io/stats/view.(*worker).start(0xc0009fc800)

[2021-12-01T09:40:42.083Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:276 +0xcd

[2021-12-01T09:40:42.083Z] created by go.opencensus.io/stats/view.init.0

[2021-12-01T09:40:42.083Z] 	/nfs/cache/mod/go.opencensus.io@v0.23.0/stats/view/worker.go:34 +0x68

[2021-12-01T09:40:42.083Z] 

[2021-12-01T09:40:42.083Z]  Goroutine 134 in state chan receive, with go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop on top of the stack:

[2021-12-01T09:40:42.083Z] goroutine 134 [chan receive]:

[2021-12-01T09:40:42.083Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc00000fe78)

[2021-12-01T09:40:42.083Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-12-01T09:40:42.083Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-12-01T09:40:42.083Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-12-01T09:40:42.083Z] 

[2021-12-01T09:40:42.083Z]  Goroutine 186 in state chan receive, with go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 186 [chan receive]:

[2021-12-01T09:40:42.084Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0004dd050)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-12-01T09:40:42.084Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-12-01T09:40:42.084Z] 

[2021-12-01T09:40:42.084Z]  Goroutine 189 in state chan receive, with go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 189 [chan receive]:

[2021-12-01T09:40:42.084Z] go.etcd.io/etcd/pkg/logutil.(*MergeLogger).outputLoop(0xc0004dd188)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:173 +0x3ac

[2021-12-01T09:40:42.084Z] created by go.etcd.io/etcd/pkg/logutil.NewMergeLogger

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/pkg/logutil/merge_logger.go:91 +0x85

[2021-12-01T09:40:42.084Z] 

[2021-12-01T09:40:42.084Z]  Goroutine 658 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 658 [IO wait]:

[2021-12-01T09:40:42.084Z] internal/poll.runtime_pollWait(0x7f18dadf20f0, 0x72, 0xffffffffffffffff)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/runtime/netpoll.go:222 +0x55

[2021-12-01T09:40:42.084Z] internal/poll.(*pollDesc).wait(0xc000d82818, 0x72, 0x8000, 0x8000, 0xffffffffffffffff)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45

[2021-12-01T09:40:42.084Z] internal/poll.(*pollDesc).waitRead(...)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92

[2021-12-01T09:40:42.084Z] internal/poll.(*FD).Read(0xc000d82800, 0xc0015f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5

[2021-12-01T09:40:42.084Z] net.(*netFD).Read(0xc000d82800, 0xc0015f0000, 0x8000, 0x8000, 0x16efdde, 0x800010601, 0x0)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/net/fd_posix.go:55 +0x4f

[2021-12-01T09:40:42.084Z] net.(*conn).Read(0xc0007c2628, 0xc0015f0000, 0x8000, 0x8000, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/net/net.go:183 +0x91

[2021-12-01T09:40:42.084Z] bufio.(*Reader).Read(0xc00186ecc0, 0xc0005af0d8, 0x9, 0x9, 0x14, 0x7f18daf68f18, 0x18)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/bufio/bufio.go:227 +0x222

[2021-12-01T09:40:42.084Z] io.ReadAtLeast(0x3f1a160, 0xc00186ecc0, 0xc0005af0d8, 0x9, 0x9, 0x9, 0xc000c0edd0, 0x1226c73, 0x467a9e2893d22)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/io/io.go:328 +0x87

[2021-12-01T09:40:42.084Z] io.ReadFull(...)

[2021-12-01T09:40:42.084Z] 	/usr/local/go/src/io/io.go:347

[2021-12-01T09:40:42.084Z] golang.org/x/net/http2.readFrameHeader(0xc0005af0d8, 0x9, 0x9, 0x3f1a160, 0xc00186ecc0, 0x0, 0x0, 0xc061eea667e95a57, 0x43de65cc8)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/golang.org/x/net@v0.0.0-20210503060351-7fd8e65b6420/http2/frame.go:237 +0x89

[2021-12-01T09:40:42.084Z] golang.org/x/net/http2.(*Framer).ReadFrame(0xc0005af0a0, 0xc001f88018, 0x5cda560, 0x0, 0x0)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/golang.org/x/net@v0.0.0-20210503060351-7fd8e65b6420/http2/frame.go:492 +0xa5

[2021-12-01T09:40:42.084Z] google.golang.org/grpc/internal/transport.(*http2Client).reader(0xc001116700)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1273 +0x185

[2021-12-01T09:40:42.084Z] created by google.golang.org/grpc/internal/transport.newHTTP2Client

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:300 +0xd51

[2021-12-01T09:40:42.084Z] 

[2021-12-01T09:40:42.084Z]  Goroutine 526 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 526 [chan receive]:

[2021-12-01T09:40:42.084Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc00018e360, 0xc0017620c0)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.084Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.084Z] 

[2021-12-01T09:40:42.084Z]  Goroutine 659 in state select, with google.golang.org/grpc/internal/transport.(*controlBuffer).get on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 659 [select]:

[2021-12-01T09:40:42.084Z] google.golang.org/grpc/internal/transport.(*controlBuffer).get(0xc001396870, 0x1, 0x0, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:395 +0xff

[2021-12-01T09:40:42.084Z] google.golang.org/grpc/internal/transport.(*loopyWriter).run(0xc00186ed80, 0x0, 0x0)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/controlbuf.go:513 +0x1dd

[2021-12-01T09:40:42.084Z] google.golang.org/grpc/internal/transport.newHTTP2Client.func3(0xc001116700)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:346 +0x7b

[2021-12-01T09:40:42.084Z] created by google.golang.org/grpc/internal/transport.newHTTP2Client

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:344 +0xf27

[2021-12-01T09:40:42.084Z] 

[2021-12-01T09:40:42.084Z]  Goroutine 609 in state select, with google.golang.org/grpc/internal/transport.(*http2Client).keepalive on top of the stack:

[2021-12-01T09:40:42.084Z] goroutine 609 [select]:

[2021-12-01T09:40:42.084Z] google.golang.org/grpc/internal/transport.(*http2Client).keepalive(0xc001116700)

[2021-12-01T09:40:42.084Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:1339 +0x174

[2021-12-01T09:40:42.085Z] created by google.golang.org/grpc/internal/transport.newHTTP2Client

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/http2_client.go:295 +0x173c

[2021-12-01T09:40:42.085Z] 

[2021-12-01T09:40:42.085Z]  Goroutine 497 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.085Z] goroutine 497 [select]:

[2021-12-01T09:40:42.085Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc0014907c0, 0x22317d755d8759a5, 0xc001a3e6c0)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.085Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.085Z] 

[2021-12-01T09:40:42.085Z]  Goroutine 622 in state select, with go.etcd.io/etcd/clientv3.(*Client).autoSync on top of the stack:

[2021-12-01T09:40:42.085Z] goroutine 622 [select]:

[2021-12-01T09:40:42.085Z] go.etcd.io/etcd/clientv3.(*Client).autoSync(0xc000970000)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/client.go:189 +0xee

[2021-12-01T09:40:42.085Z] created by go.etcd.io/etcd/clientv3.newClient

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/client.go:497 +0xbe5

[2021-12-01T09:40:42.085Z] 

[2021-12-01T09:40:42.085Z]  Goroutine 674 in state select, with google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient on top of the stack:

[2021-12-01T09:40:42.085Z] goroutine 674 [select]:

[2021-12-01T09:40:42.085Z] google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc00089c640, 0xc000529ee8, 0x5, 0x5, 0xc000c33920, 0x5, 0xc000c33950)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:186 +0xb1

[2021-12-01T09:40:42.085Z] google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc00089c640, 0xc000529ee8, 0x5, 0x5, 0x11d90aa, 0xc000b294a0, 0x3b6b4f8)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:166 +0x190

[2021-12-01T09:40:42.085Z] google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc001b97230, 0xc000529ee8, 0x5, 0x5, 0x1, 0xc000c339e8, 0x1716bbc)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:479 +0x55

[2021-12-01T09:40:42.085Z] io.ReadAtLeast(0x3f1dba0, 0xc001b97230, 0xc000529ee8, 0x5, 0x5, 0x5, 0x380c100, 0xc00077bf01, 0xc002324000)

[2021-12-01T09:40:42.085Z] 	/usr/local/go/src/io/io.go:328 +0x87

[2021-12-01T09:40:42.085Z] io.ReadFull(...)

[2021-12-01T09:40:42.085Z] 	/usr/local/go/src/io/io.go:347

[2021-12-01T09:40:42.085Z] google.golang.org/grpc/internal/transport.(*Stream).Read(0xc001942e00, 0xc000529ee8, 0x5, 0x5, 0x0, 0x0, 0x3893a40)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:463 +0xd2

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.(*parser).recvMsg(0xc000529ed8, 0x7fffffff, 0xc001d0dd88, 0xc0013a4000, 0x28, 0x28, 0x0, 0x0)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:510 +0x63

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.recvAndDecompress(0xc000529ed8, 0xc001942e00, 0x0, 0x0, 0x7fffffff, 0x0, 0x0, 0x0, 0xc0013a4000, 0x28, ...)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:641 +0x4d

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.recv(0xc000529ed8, 0x7f18b110ee48, 0x5d0d278, 0xc001942e00, 0x0, 0x0, 0x3893a40, 0xc00197a078, 0x7fffffff, 0x0, ...)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:709 +0x9b

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.(*csAttempt).recvMsg(0xc001938480, 0x3893a40, 0xc00197a078, 0x0, 0xc061eeb1a7e93397, 0x1100000eb81bb802)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:884 +0xee

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001938480, 0x1, 0x7bbf8094a)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:735 +0x46

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.(*clientStream).withRetry(0xc00075ca20, 0xc000c33e88, 0xc000c33e58, 0x18, 0x18)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:589 +0x3df

[2021-12-01T09:40:42.085Z] google.golang.org/grpc.(*clientStream).RecvMsg(0xc00075ca20, 0x3893a40, 0xc00197a078, 0x0, 0xc001922be0)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:734 +0x105

[2021-12-01T09:40:42.085Z] go.etcd.io/etcd/etcdserver/etcdserverpb.(*leaseLeaseKeepAliveClient).Recv(0xc001d01ec0, 0xc001d0dd88, 0x0, 0x0)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/etcdserver/etcdserverpb/rpc.pb.go:3835 +0x62

[2021-12-01T09:40:42.085Z] go.etcd.io/etcd/clientv3.(*lessor).recvKeepAliveLoop(0xc001922be0, 0x0, 0x0)

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:447 +0xb3

[2021-12-01T09:40:42.085Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive.func1

[2021-12-01T09:40:42.085Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:299 +0x45

[2021-12-01T09:40:42.085Z] 

[2021-12-01T09:40:42.085Z]  Goroutine 652 in state select, with github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1 on top of the stack:

[2021-12-01T09:40:42.085Z] goroutine 652 [select]:

[2021-12-01T09:40:42.085Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop.func1(0xc0019b9b00, 0x3fb0c10, 0xc000d4d000)

[2021-12-01T09:40:42.085Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1141 +0x12f

[2021-12-01T09:40:42.085Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryReportLoop

[2021-12-01T09:40:42.085Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1133 +0x2c6

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 623 in state select, with google.golang.org/grpc.newClientStream.func5 on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 623 [select]:

[2021-12-01T09:40:42.086Z] google.golang.org/grpc.newClientStream.func5(0xc00027e700, 0xc00075ca20, 0x3f69fb0, 0xc001b971a0)

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:318 +0xaa

[2021-12-01T09:40:42.086Z] created by google.golang.org/grpc.newClientStream

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:317 +0xbf0

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 621 in state chan receive, with google.golang.org/grpc.(*addrConn).resetTransport on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 621 [chan receive]:

[2021-12-01T09:40:42.086Z] google.golang.org/grpc.(*addrConn).resetTransport(0xc000f0a840)

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/clientconn.go:1156 +0x465

[2021-12-01T09:40:42.086Z] created by google.golang.org/grpc.(*addrConn).connect

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/clientconn.go:799 +0x12a

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 653 in state select, with github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1 on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 653 [select]:

[2021-12-01T09:40:42.086Z] github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop.func1(0xc0019b9b00)

[2021-12-01T09:40:42.086Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1170 +0xd8

[2021-12-01T09:40:42.086Z] created by github.com/pingcap/tidb/domain.(*Domain).TelemetryRotateSubWindowLoop

[2021-12-01T09:40:42.086Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1163 +0x7f

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 675 in state select, with go.etcd.io/etcd/clientv3.(*lessor).deadlineLoop on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 675 [select]:

[2021-12-01T09:40:42.086Z] go.etcd.io/etcd/clientv3.(*lessor).deadlineLoop(0xc001922be0)

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:539 +0x17a

[2021-12-01T09:40:42.086Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive.func1

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:300 +0x67

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 438 in state IO wait, with internal/poll.runtime_pollWait on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 438 [IO wait]:

[2021-12-01T09:40:42.086Z] internal/poll.runtime_pollWait(0x7f18dadf2008, 0x72, 0xffffffffffffffff)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/runtime/netpoll.go:222 +0x55

[2021-12-01T09:40:42.086Z] internal/poll.(*pollDesc).wait(0xc000f36098, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:87 +0x45

[2021-12-01T09:40:42.086Z] internal/poll.(*pollDesc).waitRead(...)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/internal/poll/fd_poll_runtime.go:92

[2021-12-01T09:40:42.086Z] internal/poll.(*FD).Read(0xc000f36080, 0xc0009cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/internal/poll/fd_unix.go:166 +0x1d5

[2021-12-01T09:40:42.086Z] net.(*netFD).Read(0xc000f36080, 0xc0009cb000, 0x1000, 0x1000, 0x1206f1c, 0xc000c10c38, 0x1235880)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/fd_posix.go:55 +0x4f

[2021-12-01T09:40:42.086Z] net.(*conn).Read(0xc000c23d98, 0xc0009cb000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/net.go:183 +0x91

[2021-12-01T09:40:42.086Z] net/http.(*persistConn).Read(0xc000c7b560, 0xc0009cb000, 0x1000, 0x1000, 0xc000c40b40, 0xc000c10d40, 0x11d10b5)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/http/transport.go:1922 +0x77

[2021-12-01T09:40:42.086Z] bufio.(*Reader).fill(0xc000ed0c00)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/bufio/bufio.go:101 +0x108

[2021-12-01T09:40:42.086Z] bufio.(*Reader).Peek(0xc000ed0c00, 0x1, 0x0, 0x1, 0x4, 0x1, 0x3)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/bufio/bufio.go:139 +0x4f

[2021-12-01T09:40:42.086Z] net/http.(*persistConn).readLoop(0xc000c7b560)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/http/transport.go:2083 +0x1a8

[2021-12-01T09:40:42.086Z] created by net/http.(*Transport).dialConn

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/http/transport.go:1743 +0xc77

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 439 in state select, with net/http.(*persistConn).writeLoop on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 439 [select]:

[2021-12-01T09:40:42.086Z] net/http.(*persistConn).writeLoop(0xc000c7b560)

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/http/transport.go:2382 +0xf7

[2021-12-01T09:40:42.086Z] created by net/http.(*Transport).dialConn

[2021-12-01T09:40:42.086Z] 	/usr/local/go/src/net/http/transport.go:1744 +0xc9c

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 680 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 680 [select]:

[2021-12-01T09:40:42.086Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc0016e4420, 0xc00018f0e0)

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.086Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 676 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 676 [chan receive]:

[2021-12-01T09:40:42.086Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc001a3e720, 0xc0018fa9c0)

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.086Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.086Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.086Z] 

[2021-12-01T09:40:42.086Z]  Goroutine 620 in state select, with google.golang.org/grpc.(*ccBalancerWrapper).watcher on top of the stack:

[2021-12-01T09:40:42.086Z] goroutine 620 [select]:

[2021-12-01T09:40:42.087Z] google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc0006b6fc0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:69 +0xac

[2021-12-01T09:40:42.087Z] created by google.golang.org/grpc.newCCBalancerWrapper

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/balancer_conn_wrappers.go:60 +0x172

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 4854 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 4854 [select]:

[2021-12-01T09:40:42.087Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc001efca20, 0xc00166be60, 0xc00062d3e0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.087Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 624 in state select, with go.etcd.io/etcd/clientv3.(*lessor).sendKeepAliveLoop on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 624 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*lessor).sendKeepAliveLoop(0xc001922be0, 0x3f8db38, 0xc001d01ec0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:579 +0x327

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*lessor).resetRecv

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:489 +0x227

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 679 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 679 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc0016e4370, 0xc00018f0e0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 656 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 656 [chan receive]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc00090fbc0, 0xc001760f60)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 695 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 695 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc00057ee80, 0x22317d755d8759b6, 0xc000d9c6c0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 694 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 694 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc0009c8000, 0xc00018f0e0)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 690 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).run on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 690 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).run(0xc000764000)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:530 +0x3c5

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*watcher).newWatcherGrpcStream

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:283 +0x2ff

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 655 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 655 [select]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc000d7d380, 0x22317d755d8759c4, 0xc00090fb60)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 696 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 696 [chan receive]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc000d9c720, 0xc001763a40)

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.087Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.087Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.087Z] 

[2021-12-01T09:40:42.087Z]  Goroutine 528 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.087Z] goroutine 528 [chan receive]:

[2021-12-01T09:40:42.087Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc00018ec60, 0xc001762540)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.088Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.088Z] 

[2021-12-01T09:40:42.088Z]  Goroutine 693 in state select, with google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient on top of the stack:

[2021-12-01T09:40:42.088Z] goroutine 693 [select]:

[2021-12-01T09:40:42.088Z] google.golang.org/grpc/internal/transport.(*recvBufferReader).readClient(0xc000cda370, 0xc001048358, 0x5, 0x5, 0x21, 0x2, 0x1f)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:186 +0xb1

[2021-12-01T09:40:42.088Z] google.golang.org/grpc/internal/transport.(*recvBufferReader).Read(0xc000cda370, 0xc001048358, 0x5, 0x5, 0xc000c14a28, 0xc00056cc60, 0x3b6b4f8)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:166 +0x190

[2021-12-01T09:40:42.088Z] google.golang.org/grpc/internal/transport.(*transportReader).Read(0xc0006e7740, 0xc001048358, 0x5, 0x5, 0x2c, 0xc000c14a00, 0x1716bbc)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:479 +0x55

[2021-12-01T09:40:42.088Z] io.ReadAtLeast(0x3f1dba0, 0xc0006e7740, 0xc001048358, 0x5, 0x5, 0x5, 0xc00138247e, 0x2e, 0x2e)

[2021-12-01T09:40:42.088Z] 	/usr/local/go/src/io/io.go:328 +0x87

[2021-12-01T09:40:42.088Z] io.ReadFull(...)

[2021-12-01T09:40:42.088Z] 	/usr/local/go/src/io/io.go:347

[2021-12-01T09:40:42.088Z] google.golang.org/grpc/internal/transport.(*Stream).Read(0xc00065c400, 0xc001048358, 0x5, 0x5, 0x4c, 0x1a, 0x2e)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/internal/transport/transport.go:463 +0xd2

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.(*parser).recvMsg(0xc001048348, 0x7fffffff, 0xc001670d20, 0xc001382460, 0x4c, 0x4c, 0x0, 0x0)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:510 +0x63

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.recvAndDecompress(0xc001048348, 0xc00065c400, 0x0, 0x0, 0x7fffffff, 0x0, 0x0, 0x0, 0xc001382460, 0x4c, ...)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:641 +0x4d

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.recv(0xc001048348, 0x7f18b110ee48, 0x5d0d278, 0xc00065c400, 0x0, 0x0, 0x3917480, 0xc002243f40, 0x7fffffff, 0x0, ...)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/rpc_util.go:709 +0x9b

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.(*csAttempt).recvMsg(0xc001c1e200, 0x3917480, 0xc002243f40, 0x0, 0x1206f1c, 0xc000c14d80)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:884 +0xee

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.(*clientStream).RecvMsg.func1(0xc001c1e200, 0x8, 0x11d1845)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:735 +0x46

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.(*clientStream).withRetry(0xc001b80120, 0xc000c14ea0, 0xc000c14e70, 0x50, 0x50)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:589 +0x3df

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.(*clientStream).RecvMsg(0xc001b80120, 0x3917480, 0xc002243f40, 0x2, 0x2)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:734 +0x105

[2021-12-01T09:40:42.088Z] go.etcd.io/etcd/etcdserver/etcdserverpb.(*watchWatchClient).Recv(0xc000698630, 0xc000c14f50, 0x0, 0x1)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/etcdserver/etcdserverpb/rpc.pb.go:3704 +0x62

[2021-12-01T09:40:42.088Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveWatchClient(0xc000764000, 0x3f8dbe8, 0xc000698630)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:764 +0x5c

[2021-12-01T09:40:42.088Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).newWatchClient

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:919 +0x556

[2021-12-01T09:40:42.088Z] 

[2021-12-01T09:40:42.088Z]  Goroutine 692 in state select, with google.golang.org/grpc.newClientStream.func5 on top of the stack:

[2021-12-01T09:40:42.088Z] goroutine 692 [select]:

[2021-12-01T09:40:42.088Z] google.golang.org/grpc.newClientStream.func5(0xc00027e700, 0xc001b80120, 0x3f69fb0, 0xc0006e76b0)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:318 +0xaa

[2021-12-01T09:40:42.088Z] created by google.golang.org/grpc.newClientStream

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/google.golang.org/grpc@v1.29.1/stream.go:317 +0xbf0

[2021-12-01T09:40:42.088Z] 

[2021-12-01T09:40:42.088Z]  Goroutine 657 in state select, with github.com/pingcap/tidb/owner.(*ownerManager).watchOwner on top of the stack:

[2021-12-01T09:40:42.088Z] goroutine 657 [select]:

[2021-12-01T09:40:42.088Z] github.com/pingcap/tidb/owner.(*ownerManager).watchOwner(0xc0004ba360, 0x3f69f08, 0xc000d7d3c0, 0xc0022fe0f0, 0xc001f68f30, 0x26)

[2021-12-01T09:40:42.088Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:341 +0x49e

[2021-12-01T09:40:42.088Z] github.com/pingcap/tidb/owner.(*ownerManager).campaignLoop(0xc0004ba360, 0xc0022fe0f0)

[2021-12-01T09:40:42.088Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:287 +0x99b

[2021-12-01T09:40:42.088Z] created by github.com/pingcap/tidb/owner.(*ownerManager).CampaignOwner

[2021-12-01T09:40:42.088Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:195 +0x367

[2021-12-01T09:40:42.088Z] 

[2021-12-01T09:40:42.088Z]  Goroutine 525 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.088Z] goroutine 525 [select]:

[2021-12-01T09:40:42.088Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc00057e040, 0x22317d755d8759a8, 0xc00018e0c0)

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.088Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.088Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.088Z] 

[2021-12-01T09:40:42.088Z]  Goroutine 654 in state select, with github.com/pingcap/tidb/domain.(*Domain).loadStatsWorker on top of the stack:

[2021-12-01T09:40:42.088Z] goroutine 654 [select]:

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).loadStatsWorker(0xc0019b9b00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1305 +0x3cc

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).UpdateTableStatsLoop

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1248 +0x2fa

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 527 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 527 [select]:

[2021-12-01T09:40:42.089Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc00057e3c0, 0x22317d755d8759ad, 0xc00018ec00)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.089Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 640 in state select, with github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1 on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 640 [select]:

[2021-12-01T09:40:42.089Z] github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1(0x3f69f40, 0xc0001a4010, 0xffffffffffffffff, 0x5)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/config.go:181 +0x452

[2021-12-01T09:40:42.089Z] github.com/tikv/client-go/v2/internal/retry.(*Backoffer).BackoffWithCfgAndMaxSleep(0xc000155840, 0xc0005b5980, 0xffffffffffffffff, 0x3f1bce0, 0xc000c43800, 0xc0001556b8, 0x11d9cf8)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/backoff.go:166 +0x8d3

[2021-12-01T09:40:42.089Z] github.com/tikv/client-go/v2/internal/retry.(*Backoffer).Backoff(0xc000155840, 0xc0005b5980, 0x3f1bce0, 0xc000c43800, 0x0, 0x0)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/backoff.go:120 +0xda

[2021-12-01T09:40:42.089Z] github.com/tikv/client-go/v2/tikv.(*KVStore).getTimestampWithRetry(0xc0001e5d40, 0xc000155840, 0x3a672e1, 0x6, 0x0, 0x0, 0x0)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/tikv/kv.go:391 +0x1fc

[2021-12-01T09:40:42.089Z] github.com/tikv/client-go/v2/tikv.(*KVStore).CurrentTimestamp(0xc0001e5d40, 0x3a672e1, 0x6, 0x1264426, 0x61a74316, 0x27b99372)

[2021-12-01T09:40:42.089Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/tikv/kv.go:357 +0xf0

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/store/driver.(*tikvStore).CurrentVersion(0xc00069c7d0, 0x3a672e1, 0x6, 0xc4dde, 0x0, 0xc000ed13e0)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/store/driver/tikv_driver.go:317 +0x46

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).Reload(0xc0019b9b00, 0x0, 0x0)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:375 +0xe5

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).loadSchemaInLoop(0xc0019b9b00, 0x3f69f08, 0xc0006b71c0, 0x3b9aca00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:567 +0x76a

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:826 +0x7f1

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 641 in state select, with github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 641 [select]:

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).topNSlowQueryLoop(0xc0019b9b00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:455 +0x1e5

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:829 +0x6af

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 706 in state select, with github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 706 [select]:

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).infoSyncerKeeper(0xc0019b9b00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:487 +0x138

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:830 +0x6d4

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 707 in state select, with github.com/pingcap/tidb/domain.(*Domain).renewLease on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 707 [select]:

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).renewLease(0xc0019b9b00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1739 +0xd1

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:831 +0x6f9

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.089Z]  Goroutine 708 in state select, with github.com/pingcap/tidb/domain.(*Domain).globalConfigSyncerKeeper on top of the stack:

[2021-12-01T09:40:42.089Z] goroutine 708 [select]:

[2021-12-01T09:40:42.089Z] github.com/pingcap/tidb/domain.(*Domain).globalConfigSyncerKeeper(0xc0019b9b00)

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:510 +0xf4

[2021-12-01T09:40:42.089Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.089Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:832 +0x71e

[2021-12-01T09:40:42.089Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 709 in state select, with github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 709 [select]:

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/domain.(*Domain).topologySyncerKeeper(0xc0019b9b00)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:533 +0x193

[2021-12-01T09:40:42.090Z] created by github.com/pingcap/tidb/domain.(*Domain).Init

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:835 +0x785

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 687 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 687 [select]:

[2021-12-01T09:40:42.090Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc0016e5c30, 0xc00018f0e0)

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.090Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 697 in state select, with github.com/pingcap/tidb/owner.(*ownerManager).watchOwner on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 697 [select]:

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/owner.(*ownerManager).watchOwner(0xc0015a1a70, 0x3f69f08, 0xc001d06bc0, 0xc000d70e70, 0xc000566d50, 0x25)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:341 +0x49e

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/owner.(*ownerManager).campaignLoop(0xc0015a1a70, 0xc000d70e70)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:287 +0x99b

[2021-12-01T09:40:42.090Z] created by github.com/pingcap/tidb/owner.(*ownerManager).CampaignOwner

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:195 +0x367

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 698 in state select, with github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1 on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 698 [select]:

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop.func1(0xc0019b9b00, 0x3f86248, 0xc0015a1a70)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1072 +0x157

[2021-12-01T09:40:42.090Z] created by github.com/pingcap/tidb/domain.(*Domain).globalBindHandleWorkerLoop

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1059 +0x73

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 699 in state select, with github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1 on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 699 [select]:

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop.func1(0xc0019b9b00, 0x3f86248, 0xc0015a1a70, 0x3fb0c10, 0xc00011ea00)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1107 +0xde

[2021-12-01T09:40:42.090Z] created by github.com/pingcap/tidb/domain.(*Domain).handleEvolvePlanTasksLoop

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1100 +0x8f

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 805 in state select, with github.com/pingcap/tidb/store/gcworker.(*GCWorker).start on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 805 [select]:

[2021-12-01T09:40:42.090Z] github.com/pingcap/tidb/store/gcworker.(*GCWorker).start(0xc001a10120, 0x3f69f08, 0xc000f7c2c0, 0xc00185c0f0)

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/store/gcworker/gc_worker.go:220 +0x332

[2021-12-01T09:40:42.090Z] created by github.com/pingcap/tidb/store/gcworker.(*GCWorker).Start

[2021-12-01T09:40:42.090Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/store/gcworker/gc_worker.go:116 +0xda

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 870 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 870 [select]:

[2021-12-01T09:40:42.090Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc001933760, 0xc00018f0e0)

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.090Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 780 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 780 [select]:

[2021-12-01T09:40:42.090Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc00167c420, 0xc00018f0e0)

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.090Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.090Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.090Z] 

[2021-12-01T09:40:42.090Z]  Goroutine 781 in state select, with github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1 on top of the stack:

[2021-12-01T09:40:42.090Z] goroutine 781 [select]:

[2021-12-01T09:40:42.091Z] github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop.func1(0xc0019b9b00, 0xc0006b0c78, 0x8bb2c97000, 0x3fb0c10, 0xc000c8ea00)

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:940 +0x125

[2021-12-01T09:40:42.091Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadPrivilegeLoop

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:931 +0x1f8

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 732 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 732 [select]:

[2021-12-01T09:40:42.091Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc001148bb0, 0xc00018f0e0)

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.091Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 733 in state select, with github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1 on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 733 [select]:

[2021-12-01T09:40:42.091Z] github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop.func1(0xc0019b9b00, 0xc0007c2cc0, 0x6fc23ac00, 0x3fb0c10, 0xc00011ec00)

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:990 +0x125

[2021-12-01T09:40:42.091Z] created by github.com/pingcap/tidb/domain.(*Domain).LoadSysVarCacheLoop

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:981 +0x10e

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 1300 in state select, with database/sql.(*DB).connectionOpener on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 1300 [select]:

[2021-12-01T09:40:42.091Z] database/sql.(*DB).connectionOpener(0xc002079110, 0x3f69f08, 0xc000d893c0)

[2021-12-01T09:40:42.091Z] 	/usr/local/go/src/database/sql/sql.go:1133 +0xb5

[2021-12-01T09:40:42.091Z] created by database/sql.OpenDB

[2021-12-01T09:40:42.091Z] 	/usr/local/go/src/database/sql/sql.go:740 +0x12a

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 834 in state select, with go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 834 [select]:

[2021-12-01T09:40:42.091Z] go.etcd.io/etcd/clientv3.(*lessor).keepAliveCtxCloser(0xc001922be0, 0x3f69f08, 0xc001a1a1c0, 0x22317d755d8759c2, 0xc00077a1e0)

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:330 +0xe5

[2021-12-01T09:40:42.091Z] created by go.etcd.io/etcd/clientv3.(*lessor).KeepAlive

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/lease.go:297 +0x29e

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 835 in state chan receive, with go.etcd.io/etcd/clientv3/concurrency.NewSession.func1 on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 835 [chan receive]:

[2021-12-01T09:40:42.091Z] go.etcd.io/etcd/clientv3/concurrency.NewSession.func1(0xc00077a240, 0xc0019ea3c0)

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:66 +0x75

[2021-12-01T09:40:42.091Z] created by go.etcd.io/etcd/clientv3/concurrency.NewSession

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/concurrency/session.go:64 +0x25e

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 836 in state select, with github.com/pingcap/tidb/owner.(*ownerManager).watchOwner on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 836 [select]:

[2021-12-01T09:40:42.091Z] github.com/pingcap/tidb/owner.(*ownerManager).watchOwner(0xc001b72cf0, 0x3f69f08, 0xc000f7c000, 0xc001cb4480, 0xc00185e2d0, 0x22)

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:341 +0x49e

[2021-12-01T09:40:42.091Z] github.com/pingcap/tidb/owner.(*ownerManager).campaignLoop(0xc001b72cf0, 0xc001cb4480)

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:287 +0x99b

[2021-12-01T09:40:42.091Z] created by github.com/pingcap/tidb/owner.(*ownerManager).CampaignOwner

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/owner/manager.go:195 +0x367

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 837 in state select, with github.com/pingcap/tidb/domain.(*Domain).updateStatsWorker on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 837 [select]:

[2021-12-01T09:40:42.091Z] github.com/pingcap/tidb/domain.(*Domain).updateStatsWorker(0xc0019b9b00, 0x3fb0c10, 0xc000c8f400, 0x3f86248, 0xc001b72cf0)

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1373 +0x288

[2021-12-01T09:40:42.091Z] created by github.com/pingcap/tidb/domain.(*Domain).UpdateTableStatsLoop

[2021-12-01T09:40:42.091Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1260 +0x1d6

[2021-12-01T09:40:42.091Z] 

[2021-12-01T09:40:42.091Z]  Goroutine 838 in state select, with github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1 on top of the stack:

[2021-12-01T09:40:42.091Z] goroutine 838 [select]:

[2021-12-01T09:40:42.091Z] github.com/tikv/client-go/v2/internal/retry.newBackoffFn.func1(0x3f69f40, 0xc0001a4010, 0xffffffffffffffff, 0xa)

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/config.go:181 +0x452

[2021-12-01T09:40:42.091Z] github.com/tikv/client-go/v2/internal/retry.(*Backoffer).BackoffWithCfgAndMaxSleep(0xc001350790, 0xc0005b59b0, 0xffffffffffffffff, 0x3f1b700, 0xc0018741c8, 0xc000bf2a20, 0x3f69f40)

[2021-12-01T09:40:42.091Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/backoff.go:166 +0x8d3

[2021-12-01T09:40:42.091Z] github.com/tikv/client-go/v2/internal/retry.(*Backoffer).Backoff(0xc001350790, 0xc0005b59b0, 0x3f1b700, 0xc0018741c8, 0x0, 0x0)

[2021-12-01T09:40:42.092Z] 	/nfs/cache/mod/github.com/tikv/client-go/v2@v2.0.0-alpha.0.20211118154139-b11da6307c6f/internal/retry/backoff.go:120 +0xda

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/store/driver.(*tikvStore).EtcdAddrs(0xc00069c7d0, 0x3fa0410, 0xc00069c7d0, 0x7f18a9b717e0, 0xc00069c7d0, 0x1)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/store/driver/tikv_driver.go:234 +0x285

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/domain.(*Domain).Init(0xc0016ac6c0, 0x3b9aca00, 0xc0017cc6a8, 0xb2d05e00, 0x0)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:719 +0xcd5

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.(*domainMap).Get.func1(0x3880e40, 0x383dc20, 0x7f18a9e1b768)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/tidb.go:83 +0x633

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/util.RunWithRetry(0x1e, 0x1f4, 0xc001351218, 0x18, 0x5d0f620)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/util/misc.go:65 +0x7f

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.(*domainMap).Get(0x5cd2e10, 0x3fa0410, 0xc00069c7d0, 0xc0016ac6c0, 0x0, 0x0)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/tidb.go:71 +0x1f0

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.createSessionWithOpt(0x3fa0410, 0xc00069c7d0, 0x0, 0xc001351400, 0x11daf16, 0xc001351440)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/session.go:2581 +0x59

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.createSession(0x3fa0410, 0xc00069c7d0, 0xc001351428, 0x11d29ba, 0xc001a8b1a0)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/session.go:2577 +0x3e

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.createSessionFunc.func1(0xc001351478, 0xc00135145f, 0xc001a8b1a0, 0xc001351400)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/session.go:997 +0x37

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/domain.(*sessionPool).Get(0xc001b96660, 0x3f44630, 0xc001b96660, 0x2, 0x3f175dd0d1d052fa)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:865 +0x125

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/session.(*session).ExecRestrictedStmt(0xc000c8ec00, 0x3f69f40, 0xc0001a4010, 0x3f877d8, 0xc001b701e0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/session/session.go:1439 +0xf9

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/statistics/handle.(*Handle).execRestrictedSQL.func1(0x3f69f40, 0xc0001a4010, 0x7f18b10968d8, 0xc000c8ec00, 0xc000c8ec00, 0x55eec00, 0x203000, 0x203000, 0x203000, 0x203000, ...)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:129 +0x15f

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/statistics/handle.(*Handle).withRestrictedSQLExecutor(0xc001b72c60, 0x3f69f40, 0xc0001a4010, 0xc001351a10, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:120 +0x193

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/statistics/handle.(*Handle).execRestrictedSQL(0xc001b72c60, 0x3f69f40, 0xc0001a4010, 0x3b371a5, 0x64, 0xc001ade060, 0x3, 0x3, 0x25983b0, 0xc001351af8, ...)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/statistics/handle/handle.go:124 +0xcc

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/statistics/handle.(*Handle).getAutoAnalyzeParameters(0xc001b72c60, 0xc00244ec80)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/statistics/handle/update.go:892 +0xdf

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/statistics/handle.(*Handle).HandleAutoAnalyze(0xc001b72c60, 0x3fa61b8, 0xc0020f84d0, 0x0)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/statistics/handle/update.go:934 +0x2f7

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/domain.(*Domain).autoAnalyzeWorker(0xc0019b9b00, 0x3f86248, 0xc001b72cf0)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1429 +0x1c5

[2021-12-01T09:40:42.092Z] created by github.com/pingcap/tidb/domain.(*Domain).UpdateTableStatsLoop

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1263 +0x245

[2021-12-01T09:40:42.092Z] 

[2021-12-01T09:40:42.092Z]  Goroutine 839 in state select, with github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop.func1 on top of the stack:

[2021-12-01T09:40:42.092Z] goroutine 839 [select]:

[2021-12-01T09:40:42.092Z] github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop.func1(0xc0019b9b00)

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1192 +0x11b

[2021-12-01T09:40:42.092Z] created by github.com/pingcap/tidb/domain.(*Domain).PlanReplayerLoop

[2021-12-01T09:40:42.092Z] 	/home/jenkins/agent/workspace/tidb_ghpr_integration_ddl_test/go/src/github.com/pingcap/tidb/domain/domain.go:1183 +0x5f

[2021-12-01T09:40:42.092Z] 

[2021-12-01T09:40:42.092Z]  Goroutine 812 in state select, with go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream on top of the stack:

[2021-12-01T09:40:42.092Z] goroutine 812 [select]:

[2021-12-01T09:40:42.092Z] go.etcd.io/etcd/clientv3.(*watchGrpcStream).serveSubstream(0xc000764000, 0xc001768000, 0xc00018f0e0)

[2021-12-01T09:40:42.092Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:810 +0x253

[2021-12-01T09:40:42.092Z] created by go.etcd.io/etcd/clientv3.(*watchGrpcStream).run

[2021-12-01T09:40:42.092Z] 	/nfs/cache/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20210512015243-d19fbe541bf9/clientv3/watch.go:547 +0x1619

[2021-12-01T09:40:42.092Z] 

[2021-12-01T09:40:42.092Z]  Goroutine 1296 in state select, with database/sql.(*DB).connectionOpener on top of the stack:

[2021-12-01T09:40:42.092Z] goroutine 1296 [select]:

[2021-12-01T09:40:42.092Z] database/sql.(*DB).connectionOpener(0xc002078680, 0x3f69f08, 0xc000d89040)

[2021-12-01T09:40:42.092Z] 	/usr/local/go/src/database/sql/sql.go:1133 +0xb5

[2021-12-01T09:40:42.092Z] created by database/sql.OpenDB

[2021-12-01T09:40:42.092Z] 	/usr/local/go/src/database/sql/sql.go:740 +0x12a

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 4852 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 4852 [select]:

[2021-12-01T09:40:42.093Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc001efc900, 0xc00166bd40, 0xc00062d2c0)

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.093Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 4767 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 4767 [select]:

[2021-12-01T09:40:42.093Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc002467620, 0xc00166f200, 0xc001a3ec00)

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.093Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 2014 in state select, with database/sql.(*DB).connectionOpener on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 2014 [select]:

[2021-12-01T09:40:42.093Z] database/sql.(*DB).connectionOpener(0xc00192dd40, 0x3f69f08, 0xc002472f80)

[2021-12-01T09:40:42.093Z] 	/usr/local/go/src/database/sql/sql.go:1133 +0xb5

[2021-12-01T09:40:42.093Z] created by database/sql.OpenDB

[2021-12-01T09:40:42.093Z] 	/usr/local/go/src/database/sql/sql.go:740 +0x12a

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 4781 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 4781 [select]:

[2021-12-01T09:40:42.093Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc0017c12c0, 0xc0017e6fc0, 0xc00148c900)

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.093Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 4780 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 4780 [select]:

[2021-12-01T09:40:42.093Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc0017c11a0, 0xc0017e6ea0, 0xc00148c840)

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.093Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.093Z] 

[2021-12-01T09:40:42.093Z]  Goroutine 4779 in state select, with github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1 on top of the stack:

[2021-12-01T09:40:42.093Z] goroutine 4779 [select]:

[2021-12-01T09:40:42.093Z] github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher.func1(0xc0017c10e0, 0xc0017e6d80, 0xc00148c780)

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:614 +0xa5

[2021-12-01T09:40:42.093Z] created by github.com/go-sql-driver/mysql.(*mysqlConn).startWatcher

[2021-12-01T09:40:42.093Z] 	/nfs/cache/mod/github.com/go-sql-driver/mysql@v1.6.0/connection.go:611 +0xbe

[2021-12-01T09:40:42.093Z] ]

2. What did you expect to see? (Required)

3. What did you see instead (Required)

4. What is your TiDB version? (Required)
@aytrack aytrack added type/bug The issue is confirmed as a bug. component/test sig/sql-infra SIG: SQL Infra severity/major labels Dec 2, 2021
@lcwangchao lcwangchao self-assigned this Dec 2, 2021
@lcwangchao lcwangchao mentioned this issue Dec 3, 2021
Merged
12 tasks
@github-actions
Copy link

github-actions bot commented Dec 3, 2021

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

@aytrack
Copy link
Contributor Author

aytrack commented Dec 7, 2021

there are still some goroutine leak found in DDL test, including index test, insert test, other test, update test.
in ci https://ci.pingcap.net/blue/organizations/jenkins/tidb_ghpr_integration_ddl_test/detail/tidb_ghpr_integration_ddl_test/6855/pipeline

@aytrack aytrack reopened this Dec 7, 2021
@lcwangchao lcwangchao mentioned this issue Dec 7, 2021
Merged
12 tasks
@github-actions
Copy link

github-actions bot commented Dec 7, 2021

Please check whether the issue should be labeled with 'affects-x.y' or 'fixes-x.y.z', and then remove 'needs-more-info' label.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lcwangchao lcwangchao

Labels
component/test severity/major sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
2 participants
@lcwangchao @aytrack