dm/syncer: multiple rows use downstream schema (#3308) #3953

ti-chi-bot · 2021-12-17T12:56:37Z

This is an automated cherry-pick of #3308

What problem does this PR solve?

This PR is a supplement about https://github.com/pingcap/ticdc/pull/3168 which uses downstream pk/uk just to generate where condition.There will be error when upstream and downstream hava diffrent pk/uk like example below. This PR uses downstream pk/uk in compact , causality , genSql to fix this error.

DMLS :
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3);
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4);
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2;
DML4: DELETE FROM t1 WHERE c0 = 1;
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4);

Upstream Schema : create table t1(c0 int premary key, c1 int, c2 int)

Causality Group :
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3); --> group1_sql1
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4); --> group2_sql1
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2; --> group2_sql2
DML4: DELETE FROM t1 WHERE c0 = 1; --> group1_sql2
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4); --> group1_sql3

Upstream Schema : create table t1(c0 int, c1 int, c2 int, unique key(c1, c2))
In this situation, group1_sql3 will conflict with group2_sql1 when execute sql in downstream.

What is changed and how it works?

Causality will use downstream pk/uk. There will no conflict in ex.
DML1: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 3); --> group1_sql1
DML2: INSERT INTO t1(c0, c1, c2) VALUES (2, 2, 4); --> group2_sql1
DML3: UPDATE t1 SET c1= 3, c2 = 3 WHERE c0= 2; --> group2_sql2
DML4: DELETE FROM t1 WHERE c0 = 1; --> group1_sql2
DML5: INSERT INTO t1 (c0, c1, c2) VALUES (1, 2, 4); --> group2_sql3

Check List

Tests

Unit test
Integration test
Manual test (add detailed scripts or steps below)
No code

Code changes

Has exported function/method change
Has exported variable/fields change
Has interface methods change
Has persistent data change

Side effects

Possible performance regression
Increased code complexity
Breaking backward compatibility

Related changes

Need to cherry-pick to the release branch
Need to update the documentation
Need to update key monitor metrics in both TiCDC document and official document

Release note

`None`

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot · 2021-12-17T12:56:39Z

[REVIEW NOTIFICATION]

This pull request has not been approved.

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

codecov-commenter · 2021-12-20T04:52:15Z

Codecov Report

❗ No coverage uploaded for pull request base (release-5.3@81af769). Click here to learn what that means.
The diff coverage is n/a.

@@               Coverage Diff                @@
##             release-5.3      #3953   +/-   ##
================================================
  Coverage               ?   56.8592%           
================================================
  Files                  ?        216           
  Lines                  ?      23071           
  Branches               ?          0           
================================================
  Hits                   ?      13118           
  Misses                 ?       8623           
  Partials               ?       1330

WizardXiao · 2021-12-20T06:30:16Z

/run-all-tests

…ti-chi-bot/ticdc into cherry-pick-3308-to-release-5.3

WizardXiao · 2021-12-20T07:31:44Z

/run-all-tests

WizardXiao · 2021-12-20T14:20:36Z

/run-all-tests

WizardXiao · 2021-12-21T02:42:05Z

LGTM
/assign @nongfushanquan

…ti-chi-bot/ticdc into cherry-pick-3308-to-release-5.3

WizardXiao · 2021-12-21T06:27:00Z

/run-all-tests

WizardXiao · 2021-12-21T06:53:16Z

/run-dm-integration-test

WizardXiao · 2021-12-21T08:15:09Z

/assign @nongfushanquan

WizardXiao · 2021-12-21T08:26:06Z

/merge

ti-chi-bot · 2021-12-21T08:26:08Z

@WizardXiao: /merge is only allowed for the committers, you can assign this pull request to the committer in list by filling /assign @committer in the comment to help merge this pull request.

In response to this:

/merge

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

WizardXiao · 2021-12-21T08:31:36Z

/assign @lance6716

lance6716 · 2021-12-21T08:35:32Z

/merge

ti-chi-bot · 2021-12-21T08:35:34Z

This pull request has been accepted and is ready to merge.

Commit hash: 3eb4921

WizardXiao · 2021-12-21T09:03:53Z

/run-dm-integration-test

…p#3953)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ * address comments * add comments for exported functions * fix the compiling problem * workerpool: limit the rate to output deadlock warning (#3775) (#3795) * tests(ticdc): set up the sync diff output directory correctly (#3725) (#3741) * relay(dm): use binlog name comparison (#3710) (#3712) * dm/load: fix concurrent call Loader.Status (#3459) (#3468) * cdc/sorter: make unified sorter cgroup aware (#3436) (#3439) * tz (ticdc): fix timezone error (#3887) (#3906) * pkg,cdc: do not use log package (#3902) (#3940) * *: rename repo from pingcap/ticdc to pingcap/tiflow (#3959) * http_*: add log for http api and refine the err handle logic (#2997) (#3307) * etcd_worker: batch etcd patch (#3277) (#3389) * http_api (ticdc): check --cert-allowed-cn before add server common name (#3628) (#3882) * kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#3663) * owner: fix owner tick block http request (#3490) (#3530) * dm/syncer: use downstream PK/UK to generate DML (#3168) (#3256) * dep(dm): update go-mysql (#3914) (#3934) * dm/syncer: multiple rows use downstream schema (#3308) (#3953) * errorutil,sink,syncer: add errorutil to handle ignorable error (#3264) (#3995) * dm/worker: don't exit when failed to read checkpoint in relay (#3345) (#4005) * syncer(dm): use an early location to reset binlog and open safemode (#3860) * ticdc/owner: Fix ddl special comment syntax error (#3845) (#3978) * dm/scheduler: fix inconsistent of relay status (#3474) (#4009) * owner,scheduler(cdc): fix nil pointer panic in owner scheduler (#2980) (#4007) (#4016) * config(ticdc): Fix old value configuration check for maxwell protocol (#3747) (#3783) * sink(ticdc): cherry pick sink bug fix to release 5.3 (#4083) * master(dm): clean and treat invalid load task (#4004) (#4145) * loader: fix wrong progress in query-status for loader (#4093) (#4143) close #3252 * ticdc/processor: Fix backoff base delay misconfiguration (#3992) (#4028) * dm: load table structure from dump files (#3295) (#4163) * compactor: fix duplicate entry in safemode (#3432) (#3434) (#4088) * kv(ticdc): reduce eventfeed rate limited log (#4072) (#4111) close #4006 * metrics(ticdc): add resolved ts and add changefeed to dataflow (#4038) (#4104) * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * retry(dm): align with tidb latest error message (#4172) (#4254) close #4159, close #4246 * owner(ticdc): Add bootstrap and try to fix the meta information in it (#3838) (#3865) * redolog: add a precleanup process when s3 enable (#3525) (#3878) * ddl(dm): make skipped ddl pass `SplitDDL()` (#4176) (#4227) close #4173 * cdc/sink: remove Initialize method from the sink interface (#3682) (#3765) Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> * http_api (ticdc): fix http api 'get processor' panic. (#4117) (#4123) close #3840 * sink (ticdc): fix a deadlock due to checkpointTs fall back in sinkNode (#4084) (#4099) close #4055 * cdc/sink: adjust kafka initialization logic (#3192) (#4162) * try fix conflicts. * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * fix conflicts. Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com> Co-authored-by: amyangfei <yangfei@pingcap.com> Co-authored-by: lance6716 <lance6716@gmail.com> Co-authored-by: sdojjy <sdojjy@qq.com> Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>

WizardXiao and others added 3 commits November 3, 2021 09:59

This is an automated cherry-pick of pingcap#3168

8128f38

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

commit-message: merge conflict

cab561d

This is an automated cherry-pick of pingcap#3308

0c86101

Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io>

ti-chi-bot mentioned this pull request Dec 17, 2021

dm/syncer: multiple rows use downstream schema #3308

Merged

ti-chi-bot assigned amyangfei Dec 17, 2021

ti-chi-bot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Dec 17, 2021

WizardXiao added 11 commits December 17, 2021 21:53

commit-message: fix conflict

f57b50f

commit-message: fix compile error

5751cf1

commit-message: fix conflict

6b78b5f

commit-message: add comment

195a4da

commit-message: fix error code in release

2597e7e

commit-message: fix error code order

9a50336

commit-message: fix import tiflow conflict

c4d1ce8

commit-message: fix format

118be7a

commit-message: fix format

e6d3f2c

commit-message: merge with 3168

44f4671

commit-message: fix error in make check

860b16e

WizardXiao added 2 commits December 20, 2021 15:27

commit-message: fix integration test and make check

704f88d

Merge branch 'cherry-pick-3168-to-release-5.3' of https://github.com/…

3d211b2

…ti-chi-bot/ticdc into cherry-pick-3308-to-release-5.3

WizardXiao assigned WizardXiao and unassigned amyangfei Dec 20, 2021

ti-chi-bot assigned nongfushanquan Dec 21, 2021

WizardXiao unassigned nongfushanquan Dec 21, 2021

lance6716 mentioned this pull request Dec 21, 2021

dm/worker: don't exit when failed to read checkpoint in relay (#3345) #3996

Closed

WizardXiao added 2 commits December 21, 2021 14:24

commit-message: fix conflict with 3256

1af3b9d

Merge branch 'cherry-pick-3308-to-release-5.3' of https://github.com/…

3eb4921

…ti-chi-bot/ticdc into cherry-pick-3308-to-release-5.3

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels Dec 21, 2021

ti-chi-bot assigned nongfushanquan Dec 21, 2021

nongfushanquan added cherry-pick-approved Cherry pick PR approved by release team. and removed do-not-merge/cherry-pick-not-approved labels Dec 21, 2021

ti-chi-bot assigned lance6716 Dec 21, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 21, 2021

Merge branch 'release-5.3' into cherry-pick-3308-to-release-5.3

5257d59

ti-chi-bot merged commit 4ff8384 into pingcap:release-5.3 Dec 21, 2021

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 13, 2022

dm/syncer: multiple rows use downstream schema (pingcap#3308) (pingca…

d99b5a6

…p#3953)

purelind added this to the v5.3.1 milestone Feb 24, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

dm/syncer: multiple rows use downstream schema (#3308) #3953

dm/syncer: multiple rows use downstream schema (#3308) #3953

ti-chi-bot commented Dec 17, 2021

ti-chi-bot commented Dec 17, 2021

codecov-commenter commented Dec 20, 2021 •

edited

Loading

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

ti-chi-bot commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

lance6716 commented Dec 21, 2021

ti-chi-bot commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

dm/syncer: multiple rows use downstream schema (#3308) #3953

dm/syncer: multiple rows use downstream schema (#3308) #3953

Conversation

ti-chi-bot commented Dec 17, 2021

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Dec 17, 2021

codecov-commenter commented Dec 20, 2021 • edited Loading

Codecov Report

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 20, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

ti-chi-bot commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

lance6716 commented Dec 21, 2021

ti-chi-bot commented Dec 21, 2021

WizardXiao commented Dec 21, 2021

codecov-commenter commented Dec 20, 2021 •

edited

Loading