master(dm): clean and treat invalid load task #4004

lance6716 · 2021-12-21T09:08:25Z

What problem does this PR solve?

close #3771

What is changed and how it works?

if the task in load task etcd KV is stopped, we skip it when hasLoadTaskByWorkerAndSource
so when start-task, we need to check if a transfer can resolve one load task problem

Check List

Tests

Unit test
Integration test

Code changes

Side effects

Related changes

Need to cherry-pick to the release branch

Release note

Fix a bug that stop-task during load phase will cause the source always tries to transfer to a worker

ti-chi-bot · 2021-12-21T09:08:26Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

Ehco1996
GMHDBJD

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

lance6716 · 2021-12-21T09:10:10Z

/run-dm-integration-test

dm/dm/master/scheduler/scheduler.go

dm/dm/master/server.go

Ehco1996

rest LGTM

dm/tests/load_task/run.sh

dm/dm/master/server.go

Ehco1996 · 2021-12-23T01:13:30Z

dm/tests/load_task/run.sh

+		"\"unit\": \"Sync\"" 1
+
+	# after worker1 goes online, although it has unfinished load_task1, but load_task1 is stopped so should not rebound
+	export GO_FAILPOINTS="github.com/pingcap/tiflow/dm/loader/LoadDataSlowDownByTask=return(\"load_task1\")"


do we need clean the GO_FAILPOINTS env after this test done?

I still need this task stuck in load unit so following test can use it

codecov-commenter · 2021-12-23T03:36:51Z

Codecov Report

Merging #4004 (973f6bc) into master (3873d39) will decrease coverage by 1.9243%.
The diff coverage is 69.9561%.

Flag	Coverage Δ
cdc	`58.6223% <69.9561%> (+0.3857%)`	⬆️
dm	`52.2308% <ø> (-3.8037%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

@@               Coverage Diff                @@
##             master      #4004        +/-   ##
================================================
- Coverage   57.0741%   55.1497%   -1.9244%     
================================================
  Files           478        485         +7     
  Lines         56551      59857      +3306     
================================================
+ Hits          32276      33011       +735     
- Misses        20978      23483      +2505     
- Partials       3297       3363        +66

Ehco1996

LGTM

GMHDBJD

LGTM

lance6716 · 2021-12-29T06:32:10Z

/merge

ti-chi-bot · 2021-12-29T06:32:12Z

This pull request has been accepted and is ready to merge.

Commit hash: 973f6bc

lance6716 · 2021-12-29T07:59:03Z

/run-integration-test
/run-verify

lance6716 · 2021-12-29T08:58:17Z

/run-verify

lance6716 · 2021-12-29T09:19:20Z

/run-leak-test

lance6716 · 2021-12-29T10:06:39Z

/run-verify
/run-kafka-integration-test

lance6716 · 2021-12-29T11:18:01Z

/run-dm-integration-test

ti-chi-bot · 2021-12-29T11:52:26Z

In response to a cherrypick label: new pull request created: #4145.

…#4145)

* fix the txn_batch_size metric inaccuracy bug when the sink target is MQ * address comments * add comments for exported functions * fix the compiling problem * workerpool: limit the rate to output deadlock warning (#3775) (#3795) * tests(ticdc): set up the sync diff output directory correctly (#3725) (#3741) * relay(dm): use binlog name comparison (#3710) (#3712) * dm/load: fix concurrent call Loader.Status (#3459) (#3468) * cdc/sorter: make unified sorter cgroup aware (#3436) (#3439) * tz (ticdc): fix timezone error (#3887) (#3906) * pkg,cdc: do not use log package (#3902) (#3940) * *: rename repo from pingcap/ticdc to pingcap/tiflow (#3959) * http_*: add log for http api and refine the err handle logic (#2997) (#3307) * etcd_worker: batch etcd patch (#3277) (#3389) * http_api (ticdc): check --cert-allowed-cn before add server common name (#3628) (#3882) * kvclient(ticdc): fix kvclient takes too long time to recover (#3612) (#3663) * owner: fix owner tick block http request (#3490) (#3530) * dm/syncer: use downstream PK/UK to generate DML (#3168) (#3256) * dep(dm): update go-mysql (#3914) (#3934) * dm/syncer: multiple rows use downstream schema (#3308) (#3953) * errorutil,sink,syncer: add errorutil to handle ignorable error (#3264) (#3995) * dm/worker: don't exit when failed to read checkpoint in relay (#3345) (#4005) * syncer(dm): use an early location to reset binlog and open safemode (#3860) * ticdc/owner: Fix ddl special comment syntax error (#3845) (#3978) * dm/scheduler: fix inconsistent of relay status (#3474) (#4009) * owner,scheduler(cdc): fix nil pointer panic in owner scheduler (#2980) (#4007) (#4016) * config(ticdc): Fix old value configuration check for maxwell protocol (#3747) (#3783) * sink(ticdc): cherry pick sink bug fix to release 5.3 (#4083) * master(dm): clean and treat invalid load task (#4004) (#4145) * loader: fix wrong progress in query-status for loader (#4093) (#4143) close #3252 * ticdc/processor: Fix backoff base delay misconfiguration (#3992) (#4028) * dm: load table structure from dump files (#3295) (#4163) * compactor: fix duplicate entry in safemode (#3432) (#3434) (#4088) * kv(ticdc): reduce eventfeed rate limited log (#4072) (#4111) close #4006 * metrics(ticdc): add resolved ts and add changefeed to dataflow (#4038) (#4104) * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * retry(dm): align with tidb latest error message (#4172) (#4254) close #4159, close #4246 * owner(ticdc): Add bootstrap and try to fix the meta information in it (#3838) (#3865) * redolog: add a precleanup process when s3 enable (#3525) (#3878) * ddl(dm): make skipped ddl pass `SplitDDL()` (#4176) (#4227) close #4173 * cdc/sink: remove Initialize method from the sink interface (#3682) (#3765) Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> * http_api (ticdc): fix http api 'get processor' panic. (#4117) (#4123) close #3840 * sink (ticdc): fix a deadlock due to checkpointTs fall back in sinkNode (#4084) (#4099) close #4055 * cdc/sink: adjust kafka initialization logic (#3192) (#4162) * try fix conflicts. * This is an automated cherry-pick of #4192 Signed-off-by: ti-chi-bot <ti-community-prow-bot@tidb.io> * fix conflicts. * fix conflicts. Co-authored-by: zhaoxinyu <zhaoxinyu512@gmail.com> Co-authored-by: amyangfei <yangfei@pingcap.com> Co-authored-by: lance6716 <lance6716@gmail.com> Co-authored-by: sdojjy <sdojjy@qq.com> Co-authored-by: Ling Jin <7138436+3AceShowHand@users.noreply.github.com> Co-authored-by: 3AceShowHand <jinl1037@hotmail.com>

master(dm): clean and treat invalid load task

270fb2e

ti-chi-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Dec 21, 2021

lance6716 added area/dm Issues or PRs related to DM. needs-cherry-pick-release-5.3 Should cherry pick this PR to release-5.3 branch. labels Dec 21, 2021

ti-chi-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Dec 21, 2021

add ut

25a2cc4

Ehco1996 reviewed Dec 22, 2021

View reviewed changes

dm/dm/master/scheduler/scheduler.go Outdated Show resolved Hide resolved

dm/dm/master/scheduler/scheduler.go Outdated Show resolved Hide resolved

dm/dm/master/server.go Outdated Show resolved Hide resolved

address comment

9e81612

Ehco1996 reviewed Dec 23, 2021

View reviewed changes

lance6716 added 2 commits December 23, 2021 11:17

address comment

140b7fa

Merge branch 'master' into fix-orphan-load-task

973f6bc

Ehco1996 approved these changes Dec 24, 2021

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 24, 2021

lance6716 added this to the v5.4.0 milestone Dec 28, 2021

GMHDBJD approved these changes Dec 29, 2021

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 29, 2021

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 29, 2021

Merge branch 'master' into fix-orphan-load-task

ceec07b

Merge branch 'master' into fix-orphan-load-task

611b4cf

Merge branch 'master' into fix-orphan-load-task

c2b4c08

Merge branch 'master' into fix-orphan-load-task

ce767e9

ti-chi-bot merged commit 506e005 into pingcap:master Dec 29, 2021

ti-chi-bot mentioned this pull request Dec 29, 2021

master(dm): clean and treat invalid load task (#4004) #4145

Merged

ti-chi-bot added a commit that referenced this pull request Dec 30, 2021

master(dm): clean and treat invalid load task (#4004) (#4145)

e73e411

3AceShowHand pushed a commit to ti-chi-bot/tiflow that referenced this pull request Jan 13, 2022

master(dm): clean and treat invalid load task (pingcap#4004) (pingcap…

1ba5290

…#4145)

lance6716 deleted the fix-orphan-load-task branch October 13, 2022 08:38

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

master(dm): clean and treat invalid load task #4004

master(dm): clean and treat invalid load task #4004

lance6716 commented Dec 21, 2021 •

edited

Loading

ti-chi-bot commented Dec 21, 2021 •

edited

Loading

lance6716 commented Dec 21, 2021

Ehco1996 left a comment

Ehco1996 Dec 23, 2021

lance6716 Dec 23, 2021

codecov-commenter commented Dec 23, 2021 •

edited

Loading

Ehco1996 left a comment

GMHDBJD left a comment

lance6716 commented Dec 29, 2021

ti-chi-bot commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

ti-chi-bot commented Dec 29, 2021

master(dm): clean and treat invalid load task #4004

master(dm): clean and treat invalid load task #4004

Conversation

lance6716 commented Dec 21, 2021 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Dec 21, 2021 • edited Loading

lance6716 commented Dec 21, 2021

Ehco1996 left a comment

Choose a reason for hiding this comment

Ehco1996 Dec 23, 2021

Choose a reason for hiding this comment

lance6716 Dec 23, 2021

Choose a reason for hiding this comment

codecov-commenter commented Dec 23, 2021 • edited Loading

Codecov Report

Ehco1996 left a comment

Choose a reason for hiding this comment

GMHDBJD left a comment

Choose a reason for hiding this comment

lance6716 commented Dec 29, 2021

ti-chi-bot commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

lance6716 commented Dec 29, 2021

ti-chi-bot commented Dec 29, 2021

lance6716 commented Dec 21, 2021 •

edited

Loading

ti-chi-bot commented Dec 21, 2021 •

edited

Loading

codecov-commenter commented Dec 23, 2021 •

edited

Loading