Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

drainer: can't send request correctly to pump when set compressor=gzip #1152

Closed
lichunzhu opened this issue Mar 22, 2022 · 3 comments · Fixed by #1186
Closed

drainer: can't send request correctly to pump when set compressor=gzip #1152

lichunzhu opened this issue Mar 22, 2022 · 3 comments · Fixed by #1186

Comments

@lichunzhu
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do?
    If possible, provide a recipe for reproducing the error.
    Start a TiDB cluster and set compressor=grip in drainer's config.

  2. What did you expect to see?
    Drainer can replicate data correctly.

  3. What did you see instead?
    Drainer fails to pull binlogs from pump. Pump can't read request correctly:
    pump.log:

[2022/03/22 19:29:34.971 +08:00] [DEBUG] [server.go:271] ["get PullBinlogs request"] [request="{\"startFrom\":{}}"]
[2022/03/22 19:29:34.971 +08:00] [DEBUG] [server.go:273] ["PullBinlogs request quit"] [request="{\"startFrom\":{}}"]
  1. Please provide the relate downstream type and version of drainer.
    (run drainer -V in terminal to get drainer's version)
    v5.2.3
@lichunzhu lichunzhu added type/bug This issue is a bug severity/moderate labels Mar 22, 2022
@lichunzhu
Copy link
Contributor Author

Guess it's because the pump doesn't set a decompressor.

@AmoebaProtozoa
Copy link

having similar issue on v5.2.2 after changing drainer's compressor option to "gzip"
compressor settings seems to be correctly applied but failed to find correct leader

[2022/03/29 18:08:22.919 +08:00] [INFO] [pump.go:138] ["pump create pull binlogs client"] [id=]
[2022/03/29 18:08:22.919 +08:00] [INFO] [pump.go:207] ["pump grpc compression enabled"] [id=]
[2022/03/29 18:08:22.920 +08:00] [ERROR] [pump.go:152] ["pump receive binlog failed"] [id=] [error="rpc error: code = Unknown desc = cluster ID are mismatch, 0 vs 7005077429661404245"]

@lichunzhu
Copy link
Contributor Author

After a careful check this problem is introduced in #1039

In this PR we set pump/drainer's max grpc message size to max_int:

	maxGrpcMsgSize  = int(^uint(0) >> 1)

However, when grpc decompresses messages, it will left a few buffer for this, which overflows the int64:
https://github.com/grpc/grpc-go/blob/v1.44.0/rpc_util.go#L742

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment