Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cdc: unified sorting is not working properly #246

Closed
zeminzhou opened this issue Oct 9, 2022 · 0 comments · Fixed by #247
Closed

cdc: unified sorting is not working properly #246

zeminzhou opened this issue Oct 9, 2022 · 0 comments · Fixed by #247
Labels
type/bug Something isn't working

Comments

@zeminzhou
Copy link
Contributor

Bug Report

1. Describe the bug

From metric, the preparation time for the sorting of unified sort is too long, 30G data took 7 minutes.

2. Minimal reproduce step (Required)

  1. get a tso for now
  2. put 30G data into upstream tikv
  3. create a changefeed with tso got from step1
  4. watch metric

3. What did you see instead (Required)

Sorter takes 7 minutes before output data

4. What did you expect to see? (Required)

Sorter takes less than 1 minutes before output data

5. What is your migration tool and TiKV version? (Required)

  • TiKV CDC: master
@zeminzhou zeminzhou added the type/bug Something isn't working label Oct 9, 2022
@zeminzhou zeminzhou mentioned this issue Oct 9, 2022
Merged
pingyu pushed a commit that referenced this issue Oct 9, 2022
@zeminzhou
[fix #246] fix unified sorter (#247)
af3ebdc 
* fix unified sorter

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove debug log

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add metric for the input event of puller

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove puller input metric

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add back

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
pingyu pushed a commit to pingyu/tikv-migration that referenced this issue Oct 9, 2022
@zeminzhou @pingyu
[fix tikv#246] fix unified sorter (tikv#247)
fb1716d 
* fix unified sorter

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove debug log

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add metric for the input event of puller

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove puller input metric

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add back

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
@pingyu pingyu mentioned this issue Oct 9, 2022
Merged
pingyu pushed a commit to pingyu/tikv-migration that referenced this issue Oct 9, 2022
@zeminzhou @pingyu
[fix tikv#246] fix unified sorter (tikv#247)
8eed546 
* fix unified sorter

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove debug log

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add metric for the input event of puller

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove puller input metric

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add back

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
Signed-off-by: pingyu <yuping@pingcap.com>
pingyu added a commit that referenced this issue Oct 9, 2022
@pingyu @zeminzhou
[to #246] Cherry pick #247,#249 to cdc-release-v1.0 (#252)
dba5fe8 
* [fix #246] fix unified sorter (#247)

* fix unified sorter

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove debug log

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add metric for the input event of puller

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* remove puller input metric

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

* add back

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
Signed-off-by: pingyu <yuping@pingcap.com>

* fix keyspan sink metrics (#249)

Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: pingyu <yuping@pingcap.com>

* fix gh action

Signed-off-by: pingyu <yuping@pingcap.com>

Signed-off-by: zeminzhou <zhouzemin@pingcap.com>
Signed-off-by: pingyu <yuping@pingcap.com>
Co-authored-by: zzm <zhouzemin@pingcap.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[fix #246] fix unified sorter zeminzhou/migration
1 participant
@zeminzhou