Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc owner switched and latency spikes when reload the tikv instance #6415

Open
3AceShowHand opened this issue Jul 21, 2022 · 1 comment
Open

cdc owner switched and latency spikes when reload the tikv instance #6415

3AceShowHand opened this issue Jul 21, 2022 · 1 comment
Assignees
@3AceShowHand
Labels
area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. component/owner Owner component. type/enhancement The issue or PR belongs to an enhancement.

Comments

@3AceShowHand
Copy link
Contributor

  • Deployment: 3 TiCDC, 1 TiKV, 1 TiDB, 1 PD
  • Workload: 200 tables, each 2000000 rows, sysbench oltp_update_non_index
  • 3 changefeeds, 1 mysql sink, 2 blackhole.

Around 87000 captured regions, when reload the TiKV instance, TiCDC owner switched, and checkpoint latency goes up to around 30m, and last for around 50m. This is unreasonable.

image

Expected: TiCDC should tolerate TiKV reload, the owner should not switch, and the checkpoint should not goes up too high.
@3AceShowHand 3AceShowHand added component/kv-client TiKV kv log client component. type/enhancement The issue or PR belongs to an enhancement. area/ticdc Issues or PRs related to TiCDC. component/owner Owner component. labels Jul 21, 2022
@3AceShowHand 3AceShowHand self-assigned this Jul 21, 2022
@3AceShowHand 3AceShowHand changed the title cdc latency spikes when reload the tikv instance cdc owner switched and latency spikes when reload the tikv instance Jul 21, 2022
@3AceShowHand
Copy link
Contributor Author

related issue: #3612

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@3AceShowHand 3AceShowHand

Labels
area/ticdc Issues or PRs related to TiCDC. component/kv-client TiKV kv log client component. component/owner Owner component. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@3AceShowHand