Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kv client resolves lock immediately after ticdc is up #2188

Closed
overvenus opened this issue Jun 29, 2021 · 1 comment · Fixed by #2235
Closed

kv client resolves lock immediately after ticdc is up #2188

overvenus opened this issue Jun 29, 2021 · 1 comment · Fixed by #2235
Assignees
Labels
area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. severity/moderate type/bug The issue is confirmed as a bug.

Comments

@overvenus
Copy link
Member

Bug Report

Please answer these questions before submitting your issue. Thanks!

  1. What did you do? If possible, provide a recipe for reproducing the error.

https://github.com/pingcap/ticdc/blob/b6451b6481538a92ddfdfe6de5a4901789e0b159/cdc/kv/region_worker.go#L293-L315

$ less cdc.log
[2021/06/29 15:40:31.453 +08:00] [WARN] [region_worker.go:303] 
["region not receiving resolved event from tikv or resolved ts is not pushing for too long time, try to resolve lock"] 
[regionID=1031073] 
[span="[7480000000000023ffe95f728000000000ff0021960000000000fa, 7480000000000023ffe95f728000000000ff00b70e0000000000fa)"] 
[duration=74h26m11.05s] [lastEvent=821.902081ms] [resolvedTs=425901283045015552]

$ cdc cli changefeed list
  {
    "id": "bl1",
    "summary": {
      "state": "normal",
      "tso": 425901283045015552,
      "checkpoint": "2021-06-26 13:14:19.858",
      "error": {
        "addr": "172.16.5.33:47912",
        "code": "CDC:ErrOwnerUnknown",
        "message": "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
      }
    }
  }
  1. What did you expect to see?

Do not resolve lock so quickly.

  1. What did you see instead?

  2. Versions of the cluster

Release-5.0 branch b6451b6

@overvenus overvenus added type/bug The issue is confirmed as a bug. component/kv-client TiKV kv log client component. labels Jun 29, 2021
@overvenus
Copy link
Member Author

https://github.com/pingcap/ticdc/blob/3c0be0d4c0e36c62971f5e57364223d1c8c70abb/cdc/kv/region_worker.go#L580-L597

It is possible to send checkpoint ts to rtsManager at line 590 and this leads to resolve lock when checkpoint ts is 20s before (the current resolve lock threshold).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. severity/moderate type/bug The issue is confirmed as a bug.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants