Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Data race in forward tso stream #6590

Closed
rleungx opened this issue Jun 13, 2023 · 0 comments · Fixed by #6591
Closed

Data race in forward tso stream #6590

rleungx opened this issue Jun 13, 2023 · 0 comments · Fixed by #6591
Labels
type/ci The issue is related to CI.

Comments

@rleungx
Copy link
Member

rleungx commented Jun 13, 2023

Flaky Test

Which jobs are failing

==================
WARNING: DATA RACE
Read at 0x00c001c12070 by goroutine 1238734:
  google.golang.org/grpc.(*clientStream).SendMsg()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/stream.go:691 +0x132
  github.com/pingcap/kvproto/pkg/tsopb.(*tSOTsoClient).Send()
      /home/runner/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230530111525-e4919c190b46/pkg/tsopb/tsopb.pb.go:1006 +0x54
  github.com/tikv/pd/server.(*GrpcServer).forwardTSORequestAsync()
      /home/runner/work/pd/pd/server/grpc_service.go:553 +0x3dc
  github.com/tikv/pd/server.(*GrpcServer).forwardTSORequestWithDeadLine.func1()
      /home/runner/work/pd/pd/server/grpc_service.go:523 +0x99

Previous write at 0x00c001c12070 by goroutine 1227635:
  google.golang.org/grpc.(*clientStream).CloseSend()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/stream.go:772 +0x57
  github.com/pingcap/kvproto/pkg/tsopb.(*tSOTsoClient).CloseSend()
      <autogenerated>:1 +0x49
  github.com/tikv/pd/server.(*GrpcServer).forwardTSO.func1()
      /home/runner/work/pd/pd/server/grpc_service.go:415 +0xa6
  runtime.deferreturn()
      /opt/hostedtoolcache/go/1.20.1/x64/src/runtime/panic.go:476 +0x32
  github.com/tikv/pd/server.(*GrpcServer).Tso()
      /home/runner/work/pd/pd/server/grpc_service.go:333 +0x564
  github.com/pingcap/kvproto/pkg/pdpb._PD_Tso_Handler()
      /home/runner/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230530111525-e4919c190b46/pkg/pdpb/pdpb.pb.go:9280 +0xc5
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:71 +0x20e
  github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0x206
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:74 +0x186
  go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/etcdserver/api/v3rpc/interceptor.go:237 +0x670
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:79 +0x366
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1244 +0x18d5
  google.golang.org/grpc.(*Server).handleStream()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1317 +0xf69
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722 +0xec

Goroutine 1238734 (running) created at:
  github.com/tikv/pd/server.(*GrpcServer).forwardTSORequestWithDeadLine()
      /home/runner/work/pd/pd/server/grpc_service.go:523 +0x379
  github.com/tikv/pd/server.(*GrpcServer).forwardTSO()
      /home/runner/work/pd/pd/server/grpc_service.go:473 +0xadc
  github.com/tikv/pd/server.(*GrpcServer).Tso()
      /home/runner/work/pd/pd/server/grpc_service.go:333 +0x564
  github.com/pingcap/kvproto/pkg/pdpb._PD_Tso_Handler()
      /home/runner/go/pkg/mod/github.com/pingcap/kvproto@v0.0.0-20230530111525-e4919c190b46/pkg/pdpb/pdpb.pb.go:9280 +0xc5
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:71 +0x20e
  github.com/grpc-ecosystem/go-grpc-prometheus.(*ServerMetrics).StreamServerInterceptor.func1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-prometheus@v1.2.0/server_metrics.go:121 +0x206
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1.1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:74 +0x186
  go.etcd.io/etcd/etcdserver/api/v3rpc.newStreamInterceptor.func1()
      /home/runner/go/pkg/mod/go.etcd.io/etcd@v0.5.0-alpha.5.0.20220915004622-85b640cee793/etcdserver/api/v3rpc/interceptor.go:237 +0x670
  github.com/grpc-ecosystem/go-grpc-middleware.ChainStreamServer.func1()
      /home/runner/go/pkg/mod/github.com/grpc-ecosystem/go-grpc-middleware@v1.0.1-0.20190118093823-f849b5445de4/chain.go:79 +0x366
  google.golang.org/grpc.(*Server).processStreamingRPC()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1244 +0x18d5
  google.golang.org/grpc.(*Server).handleStream()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:1317 +0xf69
  google.golang.org/grpc.(*Server).serveStreams.func1.1()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:722 +0xec

Goroutine 1227635 (finished) created at:
  google.golang.org/grpc.(*Server).serveStreams.func1()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:720 +0x17a
  google.golang.org/grpc/internal/transport.(*http2Server).operateHeaders()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/internal/transport/http2_server.go:447 +0x2589
  google.golang.org/grpc/internal/transport.(*http2Server).HandleStreams()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/internal/transport/http2_server.go:488 +0x264
  google.golang.org/grpc.(*Server).serveStreams()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:718 +0x1e7
  google.golang.org/grpc.(*Server).handleRawConn.func1()
      /home/runner/go/pkg/mod/google.golang.org/grpc@v1.26.0/server.go:679 +0x4c
==================

CI link

https://github.com/tikv/pd/actions/runs/5251232104/jobs/9485979766?pr=6557

Reason for failure (if possible)

Two goroutines may operate the forward stream at the same time.

Anything else
@rleungx rleungx added the type/ci The issue is related to CI. label Jun 13, 2023
@binshi-bing binshi-bing mentioned this issue Jun 13, 2023
Merged
ti-chi-bot bot added a commit that referenced this issue Jun 13, 2023
@binshi-bing @ti-chi-bot
mcs, tso: fix stream Send() and CloseSend() data race issue in TSO pr…
7158cb3 
…oxy (#6591)

close #6590

No need to call SendClose(), because TSO proxy will cancel the stream context which will cause the corresponding grpc stream on the server side to exit.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
rleungx pushed a commit to rleungx/pd that referenced this issue Aug 2, 2023
@binshi-bing @ti-chi-bot @rleungx
mcs, tso: fix stream Send() and CloseSend() data race issue in TSO pr…
9de713e 
…oxy (tikv#6591)

close tikv#6590

No need to call SendClose(), because TSO proxy will cancel the stream context which will cause the corresponding grpc stream on the server side to exit.

Signed-off-by: Bin Shi <binshi.bing@gmail.com>

Co-authored-by: ti-chi-bot[bot] <108142056+ti-chi-bot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/ci The issue is related to CI.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

mcs, tso, tso proxy: fix stream Send() and CloseSend() data race issue in TSO proxy binshi-bing/pd
1 participant
@rleungx