*: support disaggregated tiflash #33535

guo-shaoge · 2022-03-29T03:18:48Z

Signed-off-by: guo-shaoge shaoge1994@163.com

What problem does this PR solve?

Issue Number: close #34707

Problem Summary:

Support dispatch mppTask to tiflash_compute nodes. (store/copr: fix build batchCop in disaggregated tiflash mode #40008 updates build batchCop)
Disable cop/batchCop protocol in disaggregated tiflash mode.
For now, tidb will update its tiflash_compute store cache from PD every 10 seconds. We will use better ways in the near future.

What is changed and how it works?

Check List

Tests

Unit test
Integration test:
1. test-infra E2E case is added: https://github.com/pingcap/endless/pull/657
Manual test (add detailed scripts or steps below)
No code

Side effects

Performance regression: Consumes more CPU
Performance regression: Consumes more Memory
Breaking backward compatibility

Documentation

Affects user behaviors: Add a new config disaggregated-tiflash
Contains syntax changes
Contains variable changes
Contains experimental features
Changes MySQL compatibility

Release note

*: support disaggregated tiflash

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot · 2022-03-29T03:18:49Z

[REVIEW NOTIFICATION]

This pull request has been approved by:

XuHuaiyu
wshwsh12

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

sre-bot · 2022-03-29T03:20:08Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

sre-bot · 2022-03-29T03:30:27Z

Code Coverage Details: https://codecov.io/github/pingcap/tidb/commit/186a7a8c04f76f2b02522c1994bc554247ac3352

Signed-off-by: guo-shaoge <shaoge1994@163.com>

sre-bot · 2022-03-29T06:43:05Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

Signed-off-by: guo-shaoge <shaoge1994@163.com>

sre-bot · 2022-04-27T13:15:04Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

Signed-off-by: guo-shaoge <shaoge1994@163.com>

sre-bot · 2022-04-27T13:16:33Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

Signed-off-by: guo-shaoge <shaoge1994@163.com>

sre-bot · 2022-04-28T13:31:54Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

guo-shaoge · 2022-04-28T13:32:03Z

/hold

Signed-off-by: guo-shaoge <shaoge1994@163.com>

store/copr/mpp.go

Signed-off-by: guo-shaoge <shaoge1994@163.com>

…_mpp_req Signed-off-by: guo-shaoge <shaoge1994@163.com>

sre-bot · 2022-05-16T13:27:57Z

Please follow PR Title Format:

pkg [, pkg2, pkg3]: what is changed

Or if the count of mainly changed packages are more than 3, use

*: what is changed

After you have format title, you can leave a comment /run-check_title to recheck it

…_mpp_req Signed-off-by: guo-shaoge <shaoge1994@163.com>

Signed-off-by: guo-shaoge <shaoge1994@163.com>

planner/core/task.go

store/copr/batch_coprocessor.go

Signed-off-by: guo-shaoge <shaoge1994@163.com>

…into random_dispatch_mpp_req

wshwsh12 · 2022-12-08T13:38:05Z

domain/domain.go

+				logCount++
+				s.GetRegionCache().InvalidateTiFlashComputeStores()
+				if logCount == 60 {
+					// Print log every 60*duration seconds.


How to print the log every 60s? Or you mean 600s?

Signed-off-by: guo-shaoge <shaoge1994@163.com>

…into random_dispatch_mpp_req

guo-shaoge · 2022-12-08T16:43:45Z

/merge

ti-chi-bot · 2022-12-08T16:43:52Z

This pull request has been accepted and is ready to merge.

Commit hash: a50994d

sre-bot · 2022-12-08T17:13:17Z

TiDB MergeCI notify

🔴 Bad News! New failing [1] after this pr merged.
These new failed integration tests seem to be caused by the current PR, please try to fix these new failed integration tests, thanks!

CI Name	Result	Duration	Compare with Parent commit
idc-jenkins-ci/integration-cdc-test	🟥 failed 1, success 39, total 40	22 min	New failing
idc-jenkins-ci-tidb/mybatis-test	🔴 failed 1, success 0, total 1	4 min 9 sec	Existing failure
idc-jenkins-ci-tidb/integration-common-test	🟢 all 17 tests passed	12 min	Existing passed
idc-jenkins-ci-tidb/common-test	🟢 all 11 tests passed	10 min	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-2	🟢 all 28 tests passed	6 min 41 sec	Existing passed
idc-jenkins-ci-tidb/tics-test	🟢 all 1 tests passed	6 min 29 sec	Existing passed
idc-jenkins-ci-tidb/integration-ddl-test	🟢 all 6 tests passed	6 min 4 sec	Existing passed
idc-jenkins-ci-tidb/sqllogic-test-1	🟢 all 26 tests passed	5 min 56 sec	Existing passed
idc-jenkins-ci-tidb/integration-compatibility-test	🟢 all 1 tests passed	3 min 20 sec	Existing passed
idc-jenkins-ci-tidb/plugin-test	🟢 build success, plugin test success	4min	Existing passed

randomly dispatch req to tiflash-dev1

e9d56f7

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot added do-not-merge/needs-linked-issue release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Mar 29, 2022

distribute task randomly

56f98ca

Signed-off-by: guo-shaoge <shaoge1994@163.com>

dispatch MPPTask to TiFlash ReadNodes using consistent hash

d84a99d

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot added needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Apr 27, 2022

fix fmt

f7f0578

Signed-off-by: guo-shaoge <shaoge1994@163.com>

change usage of cacheRegion.GetTiFlashMPPStores()

259e14f

Signed-off-by: guo-shaoge <shaoge1994@163.com>

ti-chi-bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Apr 28, 2022

clean buildBatchCopTasksConsistentHash() and delete useless code

a6d4681

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge mentioned this pull request May 16, 2022

Support get addresses of all tiflash_mpp node tikv/client-go#498

Open

XuHuaiyu reviewed May 16, 2022

View reviewed changes

store/copr/mpp.go Outdated Show resolved Hide resolved

guo-shaoge mentioned this pull request May 16, 2022

support find stores with tiflash_mpp label tikv/client-go#481

Merged

guo-shaoge added 2 commits May 16, 2022 21:19

fix

a4708e0

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'master' of github.com:pingcap/tidb into random_dispatch…

d9077e0

…_mpp_req Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge requested a review from a team as a code owner May 16, 2022 13:27

ti-chi-bot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels May 16, 2022

guo-shaoge added 3 commits December 6, 2022 16:31

Merge branch 'master' of github.com:pingcap/tidb into random_dispatch…

578f487

…_mpp_req Signed-off-by: guo-shaoge <shaoge1994@163.com>

fix fmt

8ffe715

Signed-off-by: guo-shaoge <shaoge1994@163.com>

update bazel

bdd16e5

Signed-off-by: guo-shaoge <shaoge1994@163.com>

guo-shaoge changed the title ~~store/copr, kv, planner, executor: support dispatch tasks to tiflash ReadNode~~ *: support disaggregated tiflash Dec 6, 2022

ti-chi-bot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Dec 8, 2022

add case

f7e278a

Signed-off-by: guo-shaoge <shaoge1994@163.com>

XuHuaiyu reviewed Dec 8, 2022

View reviewed changes

planner/core/task.go Show resolved Hide resolved

store/copr/batch_coprocessor.go Outdated Show resolved Hide resolved

guo-shaoge added 3 commits December 8, 2022 17:56

update domain.go add recover log

8d592b8

Signed-off-by: guo-shaoge <shaoge1994@163.com>

update err msg

a51ecf6

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'master' into random_dispatch_mpp_req

835acbf

guo-shaoge requested review from XuHuaiyu and wshwsh12 December 8, 2022 12:59

guo-shaoge added 4 commits December 8, 2022 21:00

update case

d2cb9c0

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'master' into random_dispatch_mpp_req

fa6e7e3

update bazel

77ca039

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'random_dispatch_mpp_req' of github.com:guo-shaoge/tidb …

4aa16d9

…into random_dispatch_mpp_req

XuHuaiyu approved these changes Dec 8, 2022

View reviewed changes

ti-chi-bot added the status/LGT1 Indicates that a PR has LGTM 1. label Dec 8, 2022

wshwsh12 reviewed Dec 8, 2022

View reviewed changes

wshwsh12 approved these changes Dec 8, 2022

View reviewed changes

ti-chi-bot added status/LGT2 Indicates that a PR has LGTM 2. and removed status/LGT1 Indicates that a PR has LGTM 1. labels Dec 8, 2022

guo-shaoge added 2 commits December 8, 2022 22:47

fix prunt path logic

c7726a4

Signed-off-by: guo-shaoge <shaoge1994@163.com>

Merge branch 'random_dispatch_mpp_req' of github.com:guo-shaoge/tidb …

a50994d

…into random_dispatch_mpp_req

ti-chi-bot added the status/can-merge Indicates a PR has been approved by a committer. label Dec 8, 2022

ti-chi-bot merged commit 4c04abc into pingcap:master Dec 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

*: support disaggregated tiflash #33535

*: support disaggregated tiflash #33535

guo-shaoge commented Mar 29, 2022 •

edited

Loading

ti-chi-bot commented Mar 29, 2022 •

edited

Loading

sre-bot commented Mar 29, 2022

sre-bot commented Mar 29, 2022 •

edited

Loading

sre-bot commented Mar 29, 2022

sre-bot commented Apr 27, 2022

sre-bot commented Apr 27, 2022

sre-bot commented Apr 28, 2022

guo-shaoge commented Apr 28, 2022

sre-bot commented May 16, 2022

wshwsh12 Dec 8, 2022

guo-shaoge commented Dec 8, 2022

ti-chi-bot commented Dec 8, 2022

sre-bot commented Dec 8, 2022

*: support disaggregated tiflash #33535

*: support disaggregated tiflash #33535

Conversation

guo-shaoge commented Mar 29, 2022 • edited Loading

What problem does this PR solve?

What is changed and how it works?

Check List

Release note

ti-chi-bot commented Mar 29, 2022 • edited Loading

sre-bot commented Mar 29, 2022

sre-bot commented Mar 29, 2022 • edited Loading

sre-bot commented Mar 29, 2022

sre-bot commented Apr 27, 2022

sre-bot commented Apr 27, 2022

sre-bot commented Apr 28, 2022

guo-shaoge commented Apr 28, 2022

sre-bot commented May 16, 2022

wshwsh12 Dec 8, 2022

Choose a reason for hiding this comment

guo-shaoge commented Dec 8, 2022

ti-chi-bot commented Dec 8, 2022

sre-bot commented Dec 8, 2022

TiDB MergeCI notify

guo-shaoge commented Mar 29, 2022 •

edited

Loading

ti-chi-bot commented Mar 29, 2022 •

edited

Loading

sre-bot commented Mar 29, 2022 •

edited

Loading