dm/syncer: multiple rows use downstream schema #3308

WizardXiao · 2021-11-05T11:39:41Z

What problem does this PR solve?

This PR is a supplement about https://github.com/pingcap/ticdc/pull/3168 which uses downstream pk/uk just to generate where condition.There will be error when upstream and downstream hava diffrent pk/uk like example below. This PR uses downstream pk/uk in compact , causality , genSql to fix this error.

DMLS :
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3);
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4);
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2;
DML4: DELETE FROM t1 WHERE c0 = 1;
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4);

Upstream Schema : create table t1(c0 int premary key, c1 int, c2 int)

Causality Group :
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3); --> group1_sql1
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4); --> group2_sql1
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2; --> group2_sql2
DML4: DELETE FROM t1 WHERE c0 = 1; --> group1_sql2
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4); --> group1_sql3

Upstream Schema : create table t1(c0 int, c1 int, c2 int, unique key(c1, c2))
In this situation, group1_sql3 will conflict with group2_sql1 when execute sql in downstream.

What is changed and how it works?

Causality will use downstream pk/uk. There will no conflict in ex.
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3); --> group1_sql1
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4); --> group2_sql1
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2; --> group2_sql2
DML4: DELETE FROM t1 WHERE c0 = 1; --> group1_sql2
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4); --> group2_sql3

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note

`None`

ti-chi-bot · 2021-11-05T11:39:42Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Ehco1996
lichunzhu

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

…tiple-rows-use-downstream-schema

WizardXiao · 2021-11-07T04:33:22Z

/run-dm-integration-test

WizardXiao · 2021-11-07T05:25:57Z

/run-dm-compatibility-test

WizardXiao · 2021-11-07T06:12:46Z

/run-kafka-integration-test

WizardXiao · 2021-11-07T07:31:07Z

/run-integration-test

codecov-commenter · 2021-11-07T07:53:18Z

Codecov Report

Merging #3308 (57a3071) into master (24030f8) will increase coverage by 0.5450%.
The diff coverage is 69.1139%.

@@               Coverage Diff                @@
##             master      #3308        +/-   ##
================================================
+ Coverage   56.5356%   57.0807%   +0.5450%     
================================================
  Files           211        216         +5     
  Lines         22798      22985       +187     
================================================
+ Hits          12889      13120       +231     
+ Misses         8598       8523        -75     
- Partials       1311       1342        +31

Ehco1996 · 2021-11-08T02:52:24Z

/run-dm-integration-tests

GMHDBJD

rest LGTM

dm/pkg/schema/tracker.go

dm/syncer/dml.go

GMHDBJD · 2021-11-08T03:05:07Z

dm/syncer/compactor_test.go

@@ -90,6 +91,8 @@ func (s *testSyncerSuite) TestCompactJob(c *C) {
 			Length: types.UnspecifiedLength,
 		}},
 	}
+	downTi := schema.GetDownStreamTi(ti, ti)


Can we use a different tableinfo in test?

How about add some integration test in shardddl1 for compact and multiple-rows by change the origin test or add new test cases.

i think it needs add integration test for compact and multiple-rows too, i will do it later.

dm/syncer/dml.go

dm/pkg/schema/tracker.go

…tiple-rows-use-downstream-schema

…com/WizardXiao/ticdc into multiple-rows-use-downstream-schema

WizardXiao · 2021-11-08T09:40:24Z

/run-dm-integration-test

WizardXiao · 2021-11-08T10:13:14Z

/run-dm-compatibility-test

WizardXiao · 2021-11-08T10:13:39Z

/run-kafka-integration-test

WizardXiao · 2021-11-08T10:14:02Z

/run-integration-test

Ehco1996

restLGTM

dm/pkg/schema/tracker.go

…tiple-rows-use-downstream-schema

WizardXiao · 2021-11-09T03:51:44Z

/run-dm-compatibility-test

WizardXiao · 2021-11-09T06:31:01Z

/run-integration-test

WizardXiao · 2021-11-09T06:31:14Z

/run-kafka-integration-test

WizardXiao · 2021-11-09T06:44:50Z

/run-leak-test

lichunzhu

rest lgtm

dm/syncer/dml.go

WizardXiao · 2021-11-09T07:10:07Z

/run-dm-integration-test

lichunzhu · 2021-11-09T09:35:55Z

/merge

ti-chi-bot · 2021-11-09T09:35:57Z

This pull request has been accepted and is ready to merge.

Commit hash: 37a03c8

GMHDBJD · 2021-12-17T12:18:11Z

/label needs-cherry-pick-release-5.3

GMHDBJD · 2021-12-17T12:18:46Z

/run-cherry-picker

amyangfei · 2021-12-17T12:24:00Z

/run-cherry-picker

amyangfei · 2021-12-17T12:25:10Z

/cherry-pick release-5.3

amyangfei · 2021-12-17T12:56:02Z

/cherry-pick release-5.3

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-12-17T12:56:38Z

@amyangfei: new pull request created: #3953.

In response to this:

/cherry-pick release-5.3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

…p#3953)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ * address comments * add comments for exported functions * fix the compiling problem * workerpool: limit the rate to output deadlock warning (#3775) (#3795) * tests(ticdc): set up the sync diff output directory correctly (#3725) (#3741) * relay(dm): use binlog name comparison (#3710) (#3712) * dm/load: fix concurrent call Loader.Status (#3459) (#3468) * cdc/sorter: make unified sorter cgroup aware (#3436) (#3439) * tz (ticdc): fix timezone error (#3887) (#3906) * pkg,cdc: do not use log package (#3902) (#3940) * *: rename repo from pingcap/ticdc to pingcap/tiflow (#3959) * http_*: add log for http api and refine the err handle logic (#2997) (#3307) * etcd_worker: batch etcd patch (#3277) (#3389) * http_api (ticdc): check --cert-allowed-cn before add server common name (#3628) (#3882) * kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#3663) * owner: fix owner tick block http request (#3490) (#3530) * dm/syncer: use downstream PK/UK to generate DML (#3168) (#3256) * dep(dm): update go-mysql (#3914) (#3934) * dm/syncer: multiple rows use downstream schema (#3308) (#3953) * errorutil,sink,syncer: add errorutil to handle ignorable error (#3264) (#3995) * dm/worker: don't exit when failed to read checkpoint in relay (#3345) (#4005) * syncer(dm): use an early location to reset binlog and open safemode (#3860) * ticdc/owner: Fix ddl special comment syntax error (#3845) (#3978) * dm/scheduler: fix inconsistent of relay status (#3474) (#4009) * owner,scheduler(cdc): fix nil pointer panic in owner scheduler (#2980) (#4007) (#4016) * config(ticdc): Fix old value configuration check for maxwell protocol (#3747) (#3783) * sink(ticdc): cherry pick sink bug fix to release 5.3 (#4083) * master(dm): clean and treat invalid load task (#4004) (#4145) * loader: fix wrong progress in query-status for loader (#4093) (#4143) close #3252 * ticdc/processor: Fix backoff base delay misconfiguration (#3992) (#4028) * dm: load table structure from dump files (#3295) (#4163) * compactor: fix duplicate entry in safemode (#3432) (#3434) (#4088) * kv(ticdc): reduce eventfeed rate limited log (#4072) (#4111) close #4006 * metrics(ticdc): add resolved ts and add changefeed to dataflow (#4038) (#4104) * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * retry(dm): align with tidb latest error message (#4172) (#4254) close #4159, close #4246 * owner(ticdc): Add bootstrap and try to fix the meta information in it (#3838) (#3865) * redolog: add a precleanup process when s3 enable (#3525) (#3878) * ddl(dm): make skipped ddl pass `SplitDDL()` (#4176) (#4227) close #4173 * cdc/sink: remove Initialize method from the sink interface (#3682) (#3765) Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> * http_api (ticdc): fix http api 'get processor' panic. (#4117) (#4123) close #3840 * sink (ticdc): fix a deadlock due to checkpointTs fall back in sinkNode (#4084) (#4099) close #4055 * cdc/sink: adjust kafka initialization logic (#3192) (#4162) * try fix conflicts. * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * fix conflicts. Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com> Co-authored-by: amyangfei <yangfei@pingcap.com> Co-authored-by: lance6716 <lance6716@gmail.com> Co-authored-by: sdojjy <sdojjy@qq.com> Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>

commit-message: multiple rows use downstream schema

3aa1beb

ti-chi-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Nov 5, 2021

WizardXiao requested a review from GMHDBJD November 5, 2021 11:42

WizardXiao added the area/dm Issues or PRs related to DM. label Nov 5, 2021

Merge branch 'master' of https://github.com/WizardXiao/ticdc into mul…

57a3071

…tiple-rows-use-downstream-schema

Merge branch 'master' into multiple-rows-use-downstream-schema

8d5a0ac

GMHDBJD reviewed Nov 8, 2021

View reviewed changes

dm/syncer/dml.go Outdated Show resolved Hide resolved

ti-chi-bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. and removed release-note Denotes a PR that will be considered when it comes time to generate release notes. labels Nov 8, 2021

Ehco1996 reviewed Nov 8, 2021

View reviewed changes

dm/pkg/schema/tracker.go Outdated Show resolved Hide resolved

dm/pkg/schema/tracker.go Outdated Show resolved Hide resolved

dm/pkg/schema/tracker.go Outdated Show resolved Hide resolved

WizardXiao added 3 commits November 8, 2021 17:30

commit-message: make any kinds indexes well-defined

7372d62

Merge branch 'master' of https://github.com/WizardXiao/ticdc into mul…

ab3e6e9

…tiple-rows-use-downstream-schema

Merge branch 'multiple-rows-use-downstream-schema' of https://github.…

b7840fb

…com/WizardXiao/ticdc into multiple-rows-use-downstream-schema

Ehco1996 reviewed Nov 9, 2021

View reviewed changes

dm/pkg/schema/tracker.go Outdated Show resolved Hide resolved

ti-chi-bot added release-note-none Denotes a PR that doesn't merit a release note. and removed do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. labels Nov 9, 2021

Merge branch 'master' of https://github.com/WizardXiao/ticdc into mul…

285e91c

…tiple-rows-use-downstream-schema

commit-message: fix availableUKIndexList init

37a03c8

lichunzhu reviewed Nov 9, 2021

View reviewed changes

dm/syncer/dml.go Show resolved Hide resolved

Ehco1996 approved these changes Nov 9, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Nov 9, 2021

lichunzhu approved these changes Nov 9, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Nov 9, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Nov 9, 2021

ti-chi-bot added 2 commits November 9, 2021 17:36

Merge branch 'master' into multiple-rows-use-downstream-schema

6456c41

Merge branch 'master' into multiple-rows-use-downstream-schema

bef35e3

ti-chi-bot merged commit b4c6b17 into pingcap:master Nov 9, 2021

amyangfei added the needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. label Dec 17, 2021

ti-chi-bot pushed a commit to ti-chi-bot/tiflow that referenced this pull request Dec 17, 2021

This is an automated cherry-pick of pingcap#3308

0c86101

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 17, 2021

dm/syncer: multiple rows use downstream schema (#3308) #3953

Merged

ti-chi-bot added a commit that referenced this pull request Dec 21, 2021

dm/syncer: multiple rows use downstream schema (#3308) (#3953)

4ff8384

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 13, 2022

dm/syncer: multiple rows use downstream schema (pingcap#3308) (pingca…

d99b5a6

…p#3953)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dm/syncer: multiple rows use downstream schema #3308

dm/syncer: multiple rows use downstream schema #3308

WizardXiao commented Nov 5, 2021 •

edited by lance6716

Loading

ti-chi-bot commented Nov 5, 2021 •

edited

Loading

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

codecov-commenter commented Nov 7, 2021

Ehco1996 commented Nov 8, 2021

GMHDBJD left a comment

GMHDBJD Nov 8, 2021

GMHDBJD Nov 8, 2021

WizardXiao Nov 8, 2021 •

edited

Loading

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

Ehco1996 left a comment

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

lichunzhu left a comment

WizardXiao commented Nov 9, 2021

lichunzhu commented Nov 9, 2021

ti-chi-bot commented Nov 9, 2021

GMHDBJD commented Dec 17, 2021

GMHDBJD commented Dec 17, 2021

amyangfei commented Dec 17, 2021

amyangfei commented Dec 17, 2021

amyangfei commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

dm/syncer: multiple rows use downstream schema #3308

dm/syncer: multiple rows use downstream schema #3308

Conversation

WizardXiao commented Nov 5, 2021 • edited by lance6716 Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Nov 5, 2021 • edited Loading

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

WizardXiao commented Nov 7, 2021

codecov-commenter commented Nov 7, 2021

Codecov Report

Ehco1996 commented Nov 8, 2021

GMHDBJD left a comment

Choose a reason for hiding this comment

GMHDBJD Nov 8, 2021

Choose a reason for hiding this comment

GMHDBJD Nov 8, 2021

Choose a reason for hiding this comment

WizardXiao Nov 8, 2021 • edited Loading

Choose a reason for hiding this comment

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

WizardXiao commented Nov 8, 2021

Ehco1996 left a comment

Choose a reason for hiding this comment

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

WizardXiao commented Nov 9, 2021

lichunzhu left a comment

Choose a reason for hiding this comment

WizardXiao commented Nov 9, 2021

lichunzhu commented Nov 9, 2021

ti-chi-bot commented Nov 9, 2021

GMHDBJD commented Dec 17, 2021

GMHDBJD commented Dec 17, 2021

amyangfei commented Dec 17, 2021

amyangfei commented Dec 17, 2021

amyangfei commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

WizardXiao commented Nov 5, 2021 •

edited by lance6716

Loading

ti-chi-bot commented Nov 5, 2021 •

edited

Loading

WizardXiao Nov 8, 2021 •

edited

Loading