Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support global sort on S3 #45719

Closed
wjhuang2016 opened this issue Aug 1, 2023 · 0 comments · Fixed by #46979
Closed

Support global sort on S3 #45719

wjhuang2016 opened this issue Aug 1, 2023 · 0 comments · Fixed by #46979
Labels
type/enhancement The issue or PR belongs to an enhancement. type/new-feature

Comments

@wjhuang2016
Copy link
Member

wjhuang2016 commented Aug 1, 2023 •
edited
Loading

Enhancement

This PR collects some PRs to support global sort.
TiDB servers can sort a large amount of data on shared storage(S3 for now), then ingect into TiKV without overlap.
@wjhuang2016 wjhuang2016 added type/enhancement The issue or PR belongs to an enhancement. type/new-feature labels Aug 1, 2023
@lance6716 lance6716 mentioned this issue Aug 1, 2023
Merged
12 tasks
@wjhuang2016 wjhuang2016 mentioned this issue Aug 1, 2023
Merged
12 tasks
@tangenta tangenta mentioned this issue Aug 1, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 2, 2023
@lance6716
lightning: extract region job's accessing data to an interface (#45717)
412fa13 
ref #45719
@tangenta tangenta mentioned this issue Aug 3, 2023
Merged
12 tasks
@lance6716 lance6716 mentioned this issue Aug 4, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 4, 2023
@tangenta
br/lightning: add byte reader implementation (#45724)
a82c8ff 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Aug 4, 2023
@wjhuang2016
*: support concurrent write for S3 writer (#45723)
5309c2f 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Aug 7, 2023
@lance6716
lightning/external: port file.go and necessary codes (#45795)
de30804 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Aug 7, 2023
@tangenta
br/lightning: add range properties codec and kv/stats reader (#45796)
7212a5f 
ref #45719
@lance6716 lance6716 mentioned this issue Aug 7, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 8, 2023
@lance6716
lightning/external: add UT and fix a bug (#45874)
3296b5e 
ref #45719
@lance6716 lance6716 mentioned this issue Aug 8, 2023
Merged
12 tasks
@tangenta tangenta mentioned this issue Aug 14, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 15, 2023
@lance6716
lightning/external: port merge iterators (#45913)
f73b21b 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Aug 15, 2023
@tangenta
br/lightning: add kv writer for external backend (#46042)
d426bcd 
ref #45719
@lance6716 lance6716 mentioned this issue Aug 16, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 18, 2023
@lance6716
lightning: add function about write-then-read to external engine (#46157
6188778 
)

ref #45719
@lance6716 lance6716 mentioned this issue Aug 21, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 23, 2023
@lance6716
lightning: reuse dupDetector to implement in-memory ingest data (#46270)
6169cce 
ref #45719
@tangenta tangenta mentioned this issue Aug 24, 2023
Merged
12 tasks
@ywqzzy ywqzzy mentioned this issue Aug 24, 2023
Merged
12 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 24, 2023
@tangenta
br/lightning: add duplicate detect adaptor for external writer (#46387)
14745fd 
ref #45719
@lance6716 lance6716 mentioned this issue Aug 24, 2023
Merged
12 tasks
@lichunzhu lichunzhu mentioned this issue Aug 24, 2023
Merged
3 tasks
ti-chi-bot bot pushed a commit that referenced this issue Aug 25, 2023
@ywqzzy
br/lightning: add cleanUp files util (#46389)
2adb1dc 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Aug 28, 2023
@lance6716
lightning: local backend use external engine part 1 (#46406)
5ecf980 
ref #45719
@lance6716 lance6716 mentioned this issue Aug 29, 2023
Merged
12 tasks
@ywqzzy ywqzzy mentioned this issue Sep 25, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Sep 25, 2023
@ywqzzy
disttask/ddl: change memory quota for writer (#47251)
a3efafe 
ref #45719
@lance6716 lance6716 mentioned this issue Sep 27, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 7, 2023
@lance6716
external: S3 concurrent reader releases memory after it's not used (#…
fec9a5b 
…47239)

ref #45719
@lance6716 lance6716 mentioned this issue Oct 10, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 11, 2023
@lance6716
external engine: support concurrent read and split into smaller data (#…
aa99c16 
…47510)

ref #45719
@ywqzzy ywqzzy mentioned this issue Oct 12, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 16, 2023
@ywqzzy
disttask: use merge sort meta (#47599)
eafb78a 
ref #45719
@wjhuang2016 wjhuang2016 mentioned this issue Oct 16, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@wjhuang2016
*: add global sort related metric (#47485)
5ac13d9 
ref #45719
@tangenta tangenta mentioned this issue Oct 17, 2023
Merged
13 tasks
@ti-chi-bot ti-chi-bot mentioned this issue Oct 17, 2023
Merged
13 tasks
ti-chi-bot pushed a commit to ti-chi-bot/tidb that referenced this issue Oct 17, 2023
@wjhuang2016 @ti-chi-bot
*: add global sort related metric (pingcap#47485)
f538e0e 
ref pingcap#45719
ti-chi-bot bot pushed a commit that referenced this issue Oct 17, 2023
@ti-chi-bot
*: add global sort related metric and fix cancel add index (#47485) (#…
5c6b0ef 
…47710)

ref #45719, close #47715
ti-chi-bot bot pushed a commit that referenced this issue Oct 18, 2023
@tangenta
ddl: refine job_type of admin show ddl jobs (#47699)
bf2f1dd 
ref #45719
@ti-chi-bot ti-chi-bot mentioned this issue Oct 26, 2023
Closed
13 tasks
@ti-chi-bot ti-chi-bot mentioned this issue Nov 15, 2023
Merged
12 tasks
@tangenta tangenta added affects-6.5 This bug affects the 6.5.x(LTS) versions. and removed affects-6.5 This bug affects the 6.5.x(LTS) versions. labels Nov 15, 2023
@ywqzzy ywqzzy mentioned this issue Nov 20, 2023
Merged
13 tasks
ti-chi-bot bot pushed a commit that referenced this issue Nov 23, 2023
@ywqzzy
global sort: merge to one file (#48142)
9cf638f 
ref #45719
ti-chi-bot bot pushed a commit that referenced this issue Nov 24, 2023
@ti-chi-bot
*: support concurrent write for S3 writer (#45723) (#48603)
a06279f 
ref #45719, close #48607
This was referenced Dec 5, 2023
Merged
Closed
ti-chi-bot bot pushed a commit that referenced this issue Dec 6, 2023
@ti-chi-bot
*: support concurrent write for S3 writer (#45723) (#49185)
79bcffb 
ref #45719, close #48607
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@ywqzzy @wuhuizuo
disttask: use merge sort meta (pingcap#47599)
b57642a 
ref pingcap#45719
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@wjhuang2016 @wuhuizuo
*: add global sort related metric (pingcap#47485)
6b7495e 
ref pingcap#45719
wuhuizuo pushed a commit to wuhuizuo/tidb that referenced this issue Apr 2, 2024
@tangenta @wuhuizuo
ddl: refine job_type of admin show ddl jobs (pingcap#47699)
c88eb86 
ref pingcap#45719
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/enhancement The issue or PR belongs to an enhancement. type/new-feature
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

external: auto switch reader mode wjhuang2016/tidb
3 participants
@lance6716 @wjhuang2016 @tangenta