loaddata: support concurrent writing to TiKV #42667
Conversation
Signed-off-by: lance6716 <lance6716@gmail.com>
|
[REVIEW NOTIFICATION] This pull request has been approved by:
To complete the pull request process, please ask the reviewers in the list to review by filling The full list of commands accepted by this bot can be found here. Reviewer can indicate their review by submitting an approval review. |
ti-chi-bot
added
do-not-merge/needs-linked-issue
do-not-merge/work-in-progress
|
/retest |
lance6716
changed the title
ti-chi-bot
removed
the
do-not-merge/work-in-progress
|
/retest |
| @@ -300,8 +300,13 @@ func (e *LoadDataController) initFieldParams(plan *plannercore.LoadData) error { | |||
| return nil | |||
| } | |||
|
|
|||
| var ignoreInTest = false | |||
There was a problem hiding this comment.
I suggest that it is an atomic type to avoid data race report.
There was a problem hiding this comment.
will turn on race check in bazel in next commit
| encodeWorkers []*encodeWorker | ||
| commitWorkers []*commitWorker |
There was a problem hiding this comment.
prefer we make encode/commit pair into a chunk worker
There was a problem hiding this comment.
I want to separate commitWorker for future use in @you06 's concurrent INSERT INTO ... SELECT FROM
ti-chi-bot
added
status/LGT2
Signed-off-by: lance6716 <lance6716@gmail.com>
|
/merge |
|
This pull request has been accepted and is ready to merge. Commit hash: 8827c5d
|
ti-chi-bot
added
the
status/can-merge
What problem does this PR solve?
Issue Number: ref #40499
Problem Summary:
What is changed and how it works?
Check List
Tests
a simple test shows concurrent writing improves the performance indeed. tiup playground, 1G data, 4 indexes
however, during the lightning test my OS is very slow, I will deploy a cluster to test it later.
Side effects
Documentation
Release note
Please refer to Release Notes Language Style Guide to write a quality release note.