Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

Merged
merged 1 commit into from
Oct 30, 2023
Merged

[Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made #26031

merged 1 commit into from
Oct 30, 2023

Conversation

CalvinKirs
Copy link
Member

@CalvinKirs CalvinKirs commented Oct 27, 2023 •
edited
Loading

What Happend

image

mysql-load-fail-record-cleaner thread crazy increase

Proposed changes

MySqlLoad will create a new thread every time checkpoint is made, set it's only daemon thread can be create new thread

Test

image

@CalvinKirs CalvinKirs changed the title [Improve](thread)Add a unified thread name setting method [Improve](thread)Fixed the MySqlLoad will create a new thread every time checkpoint is made Oct 30, 2023
@CalvinKirs CalvinKirs changed the title [Improve](thread)Fixed the MySqlLoad will create a new thread every time checkpoint is made [Fix](MySqlLoad)Fixed the MySqlLoad will create a new thread every time checkpoint is made Oct 30, 2023
@CalvinKirs CalvinKirs force-pushed the master-test-thread-increase branch from 588b1b0 to 719bd47 Compare October 30, 2023 04:34
@CalvinKirs CalvinKirs force-pushed the master-test-thread-increase branch from 8a7f484 to 2ec8440 Compare October 30, 2023 04:43
@CalvinKirs
Copy link
Member Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.25 seconds
stream load tsv: 555 seconds loaded 74807831229 Bytes, about 128 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.0 seconds inserted 10000000 Rows, about 344K ops/s
storage size: 17162273247 Bytes

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 46.47 seconds
stream load tsv: 551 seconds loaded 74807831229 Bytes, about 129 MB/s
stream load json: 20 seconds loaded 2358488459 Bytes, about 112 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 29.2 seconds inserted 10000000 Rows, about 342K ops/s
storage size: 17162109524 Bytes

dataroaring
dataroaring approved these changes Oct 30, 2023
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 30, 2023
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit eb2cbae into apache:master Oct 30, 2023
17 of 18 checks passed
@dataroaring dataroaring added the usercase Important user case type label label Oct 30, 2023
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023
@CalvinKirs
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
e5b48e5 
…t mysql load (apache#26031)

Add a unified thread name setting method

(cherry picked from commit eb2cbae)
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023
@CalvinKirs
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
4408526 
…t mysql load (apache#26031)

Add a unified thread name setting method

(cherry picked from commit eb2cbae)
CalvinKirs added a commit to CalvinKirs/incubator-doris that referenced this pull request Oct 31, 2023
@CalvinKirs
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
ee7a524 
…t mysql load (apache#26031)

Add a unified thread name setting method

(cherry picked from commit eb2cbae)
xiaokang pushed a commit that referenced this pull request Oct 31, 2023
@CalvinKirs
(pick-2.0.2.1)[Fix](MySqlLoad)Fixed the MySqlLoad will create a new t…
ab3de52 
…hread every time checkpoint is made #26031 (#26136)
@morrySnow morrySnow mentioned this pull request Oct 31, 2023
Merged
morrySnow added a commit that referenced this pull request Nov 1, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (#…
18dabe7 
…26164)

1. FE could not compile because below error. Intro by PR #25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - #25951
> - #26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR #25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (#26084)"
> This reverts commit 0d956e9.
xiaokang pushed a commit that referenced this pull request Nov 1, 2023
@CalvinKirs
(pick-2.0-26031)[Fix](MySqlLoad) Fix meaningless thread creation ever…
0242f87 
…y time checkpoint mysql load (#26031) (#26139)
@xiaokang xiaokang mentioned this pull request Nov 1, 2023
Merged
dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
@CalvinKirs
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
c2977cf 
…t mysql load (apache#26031)

Add a unified thread name setting method
dutyu pushed a commit to dutyu/doris that referenced this pull request Nov 4, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
0ee2870 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
@CalvinKirs @seawinde
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
18207c5 
…t mysql load (apache#26031)

Add a unified thread name setting method
seawinde pushed a commit to seawinde/doris that referenced this pull request Nov 13, 2023
@morrySnow @seawinde
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
4677a82 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
gnehil pushed a commit to gnehil/doris that referenced this pull request Dec 4, 2023
@CalvinKirs
(pick-2.0-26031)[Fix](MySqlLoad) Fix meaningless thread creation ever…
7b2e5b9 
…y time checkpoint mysql load (apache#26031) (apache#26139)
@xiaokang xiaokang mentioned this pull request Dec 4, 2023
Open
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@CalvinKirs
[Fix](MySqlLoad) Fix meaningless thread creation every time checkpoin…
2994f13 
…t mysql load (apache#26031)

Add a unified thread name setting method
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@morrySnow
[fix](compile) fe compile failed when generate doc and FE UT failed (a…
ae8163e 
…pache#26164)

1. FE could not compile because below error. Intro by PR apache#25933
```
[INFO] --- exec:3.1.0:java (doc) @ fe-core ---
...
Failed to generate doc for ignoreRuntimeFilterIds
```

2. fix UT bugs intro by below PRs
> - apache#25951
> - apache#26031

3. because fe could not compile, FE UT CI do not work well. So, some UT failed be introduced by the PRs merged after PR apache#25933 merged. So this PR revert them to fix FE UT

> - Revert "[Bug](materialized-view) SelectMaterializedIndexWithAggregate do not change plan > when match ba… (apache#26145)"
> This reverts commit 8d7abf6.

> - Revert "[enhancement](Nereids): optimize GroupExpressionMatching (apache#26130)"
> This reverts commit 19122b5.

> - Revert "[Performance](Nereids): optimize GroupExpressionMatching (apache#26084)"
> This reverts commit 0d956e9.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@bobhan1 bobhan1 bobhan1 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.3-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@CalvinKirs @doris-robot @bobhan1 @dataroaring @xiaokang @wm1581066