lightning: provide view for users when using conflict detection #53289

lyzx2001 · 2024-05-15T06:21:25Z

What problem does this PR solve?

Issue Number: close #52393

Problem Summary:

What changed and how does it work?

Provide view for users regardless of the lightning configuration.

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No need to test
- I checked and no code files have been changed.

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Release note

Please refer to Release Notes Language Style Guide to write a quality release note.

For lightning conflict detection "replace" mode, the conflicting data are recorded in the `lightning_task_info.conflict_view` view of the target TiDB cluster.
If the value for column is_precheck_conflict is 0, it stands for conflicting data detected by post-import conflict detection in the physical import mode; If the value for column is_precheck_conflict is 1, it stands for conflicting data detected by preprocess conflict detection in both logical and physical import modes.

tiprow · 2024-05-15T06:21:39Z

Hi @lyzx2001. Thanks for your PR.

PRs from untrusted users cannot be marked as trusted with /ok-to-test in this repo meaning untrusted PR authors can never trigger tests themselves. Collaborators can still trigger tests on the PR using /test all.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lyzx2001 · 2024-05-15T06:21:44Z

/test pull-lightning-integration-test

tiprow · 2024-05-15T06:22:06Z

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

codecov · 2024-05-15T06:45:50Z

Codecov Report

Attention: Patch coverage is 57.14286% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 73.8923%. Comparing base (fa94f49) to head (c1b6312).
Report is 65 commits behind head on master.

Additional details and impacted files

@@               Coverage Diff                @@
##             master     #53289        +/-   ##
================================================
+ Coverage   72.4109%   73.8923%   +1.4814%     
================================================
  Files          1493       1546        +53     
  Lines        429341     453630     +24289     
================================================
+ Hits         310890     335198     +24308     
+ Misses        99162      97597      -1565     
- Partials      19289      20835      +1546

Flag	Coverage Δ
integration	`50.6486% <57.1428%> (?)`
unit	`71.3458% <28.5714%> (+0.0429%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components	Coverage Δ
dumpling	`53.9957% <ø> (ø)`
parser	`∅ <ø> (∅)`
br	`42.0730% <ø> (+0.6361%)`	⬆️

lyzx2001 · 2024-05-17T03:44:08Z

/test pull-lightning-integration-test

tiprow · 2024-05-17T03:44:28Z

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lyzx2001 · 2024-05-17T06:49:51Z

/test pull-lightning-integration-test

tiprow · 2024-05-17T06:50:12Z

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/test pull-lightning-integration-test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

lyzx2001 · 2024-05-17T08:30:52Z

/cc @lance6716 @D3Hunter

D3Hunter

rest lgtm

D3Hunter · 2024-05-20T02:39:49Z

lightning/tidb-lightning.toml

+#            You can manually insert the correct records into the target table based on your business requirements.
+#            Note that the target TiKV must be v5.2.0 or later versions; otherwise it falls back to 'none'.
+# The default value is 'none'.
+# duplicate-resolution = 'none'


recovering old config? there is a record option

I deleted the old config in #51710
But we should mark them as deprecated rather than directly delete them, so I add them back here marked as deprecated.
https://docs.pingcap.com/tidb/v8.0/tidb-lightning-configuration
The doc also marks them as deprecated rather than directly delete them.

The record option was only for internal use and was not declared in the older documentation for users, so I think we do not need to add it here either.
https://docs.pingcap.com/tidb/stable/tidb-lightning-configuration

"deprecated" means the configuration value is not recommended but still takes effect. But duplicate-resolution = "remove" does not take effect.

lyzx2001 · 2024-05-20T04:51:42Z

/cc @yudongusa @easonn7

ti-chi-bot · 2024-05-20T04:51:44Z

@lyzx2001: GitHub didn't allow me to request PR reviews from the following users: yudongusa, easonn7.

Note that only pingcap members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @yudongusa @easonn7

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

lance6716

rest lgtm

lance6716 · 2024-05-20T06:23:36Z

lightning/tidb-lightning.toml

+#            You can manually insert the correct records into the target table based on your business requirements.
+#            Note that the target TiKV must be v5.2.0 or later versions; otherwise it falls back to 'none'.
+# The default value is 'none'.
+# duplicate-resolution = 'none'


"deprecated" means the configuration value is not recommended but still takes effect. But duplicate-resolution = "remove" does not take effect.

lyzx2001 · 2024-05-20T07:53:11Z

rest lgtm

modified

lightning/tidb-lightning.toml

ti-chi-bot · 2024-05-21T06:31:04Z

[LGTM Timeline notifier]

Timeline:

2024-05-20 02:41:51.923139512 +0000 UTC m=+2053065.680275085: ☑️ agreed by D3Hunter.
2024-05-21 06:31:03.92279727 +0000 UTC m=+2153217.679932843: ☑️ agreed by lance6716.

lyzx2001 · 2024-05-21T07:02:11Z

/retest

tiprow · 2024-05-21T07:02:33Z

@lyzx2001: Cannot trigger testing until a trusted user reviews the PR and leaves an /ok-to-test message.

In response to this:

/retest

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

easonn7 · 2024-05-21T08:40:08Z

/approve

ti-chi-bot · 2024-05-21T08:40:16Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: D3Hunter, easonn7, lance6716

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~lightning/OWNERS~~ [D3Hunter,easonn7,lance6716]
~~pkg/lightning/OWNERS~~ [D3Hunter,lance6716]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

…cap#53289) close pingcap#52393

lyzx2001 added 2 commits May 13, 2024 15:38

revise tidb-lightning.toml

8f79f8f

lightning: provide view for users when using conflict detection

ddc7019

fix IT

a3a245a

lyzx2001 changed the title ~~[WIP] lightning: provide view for users when using conflict detection~~ lightning: provide view for users when using conflict detection May 17, 2024

ti-chi-bot bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 17, 2024

fix UT

803be36

ti-chi-bot bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels May 17, 2024

ti-chi-bot bot requested review from D3Hunter and lance6716 May 17, 2024 08:30

lyzx2001 mentioned this pull request May 17, 2024

update lightning conflict detection recording view pingcap/docs#17532

Merged

16 tasks

D3Hunter approved these changes May 20, 2024

View reviewed changes

ti-chi-bot bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label May 20, 2024

lance6716 reviewed May 20, 2024

View reviewed changes

revise tidb-lightning.toml

28d987c

lyzx2001 requested a review from lance6716 May 20, 2024 07:53

lance6716 reviewed May 20, 2024

View reviewed changes

lightning/tidb-lightning.toml Outdated Show resolved Hide resolved

lyzx2001 added 2 commits May 21, 2024 13:54

revise tidb-lightning.toml

588b1cc

update

38000d8

lance6716 approved these changes May 21, 2024

View reviewed changes

ti-chi-bot bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels May 21, 2024

update

c1b6312

ti-chi-bot bot added the approved label May 21, 2024

ti-chi-bot bot merged commit 67531d0 into pingcap:master May 21, 2024
22 checks passed

RidRisR pushed a commit to RidRisR/tidb that referenced this pull request May 23, 2024

lightning: provide view for users when using conflict detection (ping…

1939616

…cap#53289) close pingcap#52393

qiancai mentioned this pull request Jun 11, 2024

update lightning conflict detection recording view pingcap/docs-cn#17648

Merged

16 tasks

fubinzh mentioned this pull request Jun 27, 2024

CDC changefeed should not fail after lightning import pingcap/tiflow#11356

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lightning: provide view for users when using conflict detection #53289

lightning: provide view for users when using conflict detection #53289

lyzx2001 commented May 15, 2024

tiprow bot commented May 15, 2024

lyzx2001 commented May 15, 2024

tiprow bot commented May 15, 2024

codecov bot commented May 15, 2024 •

edited

Loading

lyzx2001 commented May 17, 2024

tiprow bot commented May 17, 2024

lyzx2001 commented May 17, 2024

tiprow bot commented May 17, 2024

lyzx2001 commented May 17, 2024

D3Hunter left a comment

D3Hunter May 20, 2024

lyzx2001 May 20, 2024

lance6716 May 20, 2024

lyzx2001 commented May 20, 2024

ti-chi-bot bot commented May 20, 2024

lance6716 left a comment

lance6716 May 20, 2024

lyzx2001 commented May 20, 2024

ti-chi-bot bot commented May 21, 2024

lyzx2001 commented May 21, 2024

tiprow bot commented May 21, 2024

easonn7 commented May 21, 2024

ti-chi-bot bot commented May 21, 2024

lightning: provide view for users when using conflict detection #53289

lightning: provide view for users when using conflict detection #53289

Conversation

lyzx2001 commented May 15, 2024

What problem does this PR solve?

What changed and how does it work?

Check List

Release note

tiprow bot commented May 15, 2024

lyzx2001 commented May 15, 2024

tiprow bot commented May 15, 2024

codecov bot commented May 15, 2024 • edited Loading

Codecov Report

lyzx2001 commented May 17, 2024

tiprow bot commented May 17, 2024

lyzx2001 commented May 17, 2024

tiprow bot commented May 17, 2024

lyzx2001 commented May 17, 2024

D3Hunter left a comment

Choose a reason for hiding this comment

D3Hunter May 20, 2024

Choose a reason for hiding this comment

lyzx2001 May 20, 2024

Choose a reason for hiding this comment

lance6716 May 20, 2024

Choose a reason for hiding this comment

lyzx2001 commented May 20, 2024

ti-chi-bot bot commented May 20, 2024

lance6716 left a comment

Choose a reason for hiding this comment

lance6716 May 20, 2024

Choose a reason for hiding this comment

lyzx2001 commented May 20, 2024

ti-chi-bot bot commented May 21, 2024

[LGTM Timeline notifier]

lyzx2001 commented May 21, 2024

tiprow bot commented May 21, 2024

easonn7 commented May 21, 2024

ti-chi-bot bot commented May 21, 2024

codecov bot commented May 15, 2024 •

edited

Loading