Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)When drop MTMV, no longer wait for task cancel to complete #45995

Merged
merged 1 commit into from
Dec 26, 2024
Merged

[enhance](mtmv)When drop MTMV, no longer wait for task cancel to complete #45995

merged 1 commit into from
Dec 26, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Dec 26, 2024 •
edited by Jibing-Li
Loading

What problem does this PR solve?

problem:

  • when drop db, will hold write lock of catalog, and drop all MTMV
  • when drop MTMV, will drop Job,
  • when drop Job, will cancel all tasks in this Job
  • when cancel task, if task is running insert overwrite, will wait for a long time

fix:
when drop job, not wait task cacncelled complete

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Dec 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 26, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@zddr
Copy link
Contributor Author

zddr commented Dec 26, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32916 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7d267cd96b94a6d6f0a453065e4c8dadb1c3584, data reload: false

------ Round 1 ----------------------------------
q1	17632	6164	6052	6052
q2	2044	294	169	169
q3	10427	1271	739	739
q4	10218	887	446	446
q5	7516	2189	2021	2021
q6	203	175	153	153
q7	880	743	611	611
q8	9236	1357	1203	1203
q9	5397	4922	4949	4922
q10	6780	2301	1871	1871
q11	474	286	269	269
q12	364	364	217	217
q13	17762	3586	3024	3024
q14	242	248	206	206
q15	554	507	492	492
q16	641	622	589	589
q17	575	853	328	328
q18	7030	6479	6437	6437
q19	1229	967	571	571
q20	317	327	189	189
q21	2899	2259	2085	2085
q22	368	335	322	322
Total cold run time: 102788 ms
Total hot run time: 32916 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6171	6309	6276	6276
q2	240	327	233	233
q3	2236	2658	2323	2323
q4	1408	1828	1361	1361
q5	4350	4763	4661	4661
q6	176	171	142	142
q7	1998	1864	1754	1754
q8	2513	2694	2647	2647
q9	6954	6810	6901	6810
q10	2904	3250	2718	2718
q11	571	507	503	503
q12	633	739	540	540
q13	3171	3634	3024	3024
q14	279	312	254	254
q15	546	507	494	494
q16	654	675	651	651
q17	1195	1705	1212	1212
q18	7299	7322	7111	7111
q19	815	1120	1065	1065
q20	1894	1985	1833	1833
q21	5582	5194	4969	4969
q22	615	649	595	595
Total cold run time: 52204 ms
Total hot run time: 51176 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 190387 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7d267cd96b94a6d6f0a453065e4c8dadb1c3584, data reload: false

query1	980	383	375	375
query2	6532	2373	2302	2302
query3	6709	214	213	213
query4	33596	23535	23602	23535
query5	4400	637	466	466
query6	295	214	188	188
query7	4617	502	307	307
query8	310	272	236	236
query9	9657	2761	2753	2753
query10	472	318	259	259
query11	18463	15388	15258	15258
query12	162	110	110	110
query13	1761	565	409	409
query14	12574	7442	7035	7035
query15	237	199	195	195
query16	8174	586	455	455
query17	1569	740	580	580
query18	2104	411	337	337
query19	227	175	151	151
query20	116	113	111	111
query21	221	119	107	107
query22	4511	4639	4330	4330
query23	34501	33152	33226	33152
query24	5995	2270	2355	2270
query25	469	457	380	380
query26	1202	265	155	155
query27	1977	459	323	323
query28	5373	2487	2438	2438
query29	705	563	402	402
query30	223	175	148	148
query31	1015	870	823	823
query32	87	77	59	59
query33	501	345	282	282
query34	749	830	503	503
query35	802	812	750	750
query36	1009	1043	958	958
query37	108	94	77	77
query38	4223	4126	4165	4126
query39	1477	1457	1434	1434
query40	200	112	99	99
query41	47	45	43	43
query42	123	105	104	104
query43	524	521	483	483
query44	1317	806	818	806
query45	175	170	168	168
query46	876	1044	664	664
query47	1930	1912	1884	1884
query48	396	412	315	315
query49	777	471	388	388
query50	628	643	383	383
query51	7124	7240	7107	7107
query52	100	98	88	88
query53	227	256	187	187
query54	480	477	405	405
query55	82	91	83	83
query56	259	261	234	234
query57	1214	1192	1155	1155
query58	234	221	225	221
query59	3064	3116	2985	2985
query60	272	262	280	262
query61	107	108	107	107
query62	856	786	757	757
query63	229	199	193	193
query64	4338	1010	649	649
query65	3268	3189	3241	3189
query66	1037	462	320	320
query67	15958	15839	15451	15451
query68	8577	750	531	531
query69	468	289	251	251
query70	1244	1152	1131	1131
query71	441	299	252	252
query72	5818	3919	3872	3872
query73	650	743	359	359
query74	10282	9019	9088	9019
query75	4692	3156	2657	2657
query76	4735	1205	791	791
query77	901	362	284	284
query78	10043	10269	9417	9417
query79	3431	925	591	591
query80	697	536	498	498
query81	466	271	234	234
query82	658	151	124	124
query83	195	172	143	143
query84	282	90	78	78
query85	783	374	306	306
query86	347	311	292	292
query87	4369	4421	4290	4290
query88	4561	2296	2204	2204
query89	405	336	291	291
query90	1916	187	188	187
query91	135	130	105	105
query92	62	56	53	53
query93	1591	880	544	544
query94	658	394	279	279
query95	325	257	254	254
query96	494	613	287	287
query97	2749	2841	2701	2701
query98	230	204	202	202
query99	1639	1617	1438	1438
Total cold run time: 297789 ms
Total hot run time: 190387 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7d267cd96b94a6d6f0a453065e4c8dadb1c3584, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.03
query3	0.24	0.08	0.08
query4	1.59	0.10	0.10
query5	0.41	0.41	0.41
query6	1.16	0.66	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.50	0.51
query10	0.55	0.58	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.60	0.61	0.60
query14	2.71	2.77	2.73
query15	0.89	0.82	0.83
query16	0.40	0.39	0.38
query17	1.06	1.05	1.06
query18	0.21	0.21	0.21
query19	1.91	1.80	2.01
query20	0.02	0.01	0.01
query21	15.36	0.91	0.60
query22	0.75	0.82	0.69
query23	15.25	1.40	0.58
query24	2.99	1.75	1.79
query25	0.25	0.16	0.11
query26	0.18	0.14	0.13
query27	0.04	0.04	0.06
query28	14.40	1.49	1.05
query29	12.63	3.97	3.27
query30	0.26	0.10	0.07
query31	2.82	0.58	0.38
query32	3.23	0.55	0.46
query33	3.23	3.19	3.14
query34	16.70	5.12	4.55
query35	4.53	4.47	4.49
query36	0.64	0.49	0.49
query37	0.09	0.06	0.06
query38	0.04	0.05	0.03
query39	0.03	0.03	0.02
query40	0.17	0.13	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.02
query43	0.04	0.04	0.03
Total cold run time: 106.54 s
Total hot run time: 32.29 s

@zddr
Copy link
Contributor Author

zddr commented Dec 26, 2024

run p0

@Jibing-Li Jibing-Li merged commit f4bff84 into apache:master Dec 26, 2024
32 of 34 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 26, 2024
@zddr
[enhance](mtmv)When drop MTMV, no longer wait for task cancel to comp…
d0ff441 
…lete (#45995)

### What problem does this PR solve?

problem:
- when drop db, will hold write lock of catalog, and drop all MTMV 
- when drop MTMV, will drop Job, 
- when drop Job, will cancel all tasks in this Job
- when cancel task, if task is running insert overwrite, will wait for a
long time

fix:
when drop job, not wait task cacncelled complete

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None
@github-actions github-actions bot mentioned this pull request Dec 26, 2024
Merged
zddr added a commit to zddr/incubator-doris that referenced this pull request Dec 26, 2024
@zddr
[enhance](mtmv)When drop MTMV, no longer wait for task cancel to comp…
36b84d2 
…lete (apache#45995)

problem:
- when drop db, will hold write lock of catalog, and drop all MTMV
- when drop MTMV, will drop Job,
- when drop Job, will cancel all tasks in this Job
- when cancel task, if task is running insert overwrite, will wait for a
long time

fix:
when drop job, not wait task cacncelled complete

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

None
@zddr zddr mentioned this pull request Dec 26, 2024
Merged
Jibing-Li pushed a commit that referenced this pull request Dec 26, 2024
@zddr
[enhance](mtmv)When drop MTMV, no longer wait for task cancel to comp…
7d5e8a1 
…lete (#45995) (#46025)

pick: #45995
Jibing-Li pushed a commit that referenced this pull request Dec 26, 2024
@github-actions @zddr
branch-3.0: [enhance](mtmv)When drop MTMV, no longer wait for task ca…
9a49926 
…ncel to complete #45995 (#46014)

Cherry-picked from #45995

Co-authored-by: zhangdong <zhangdong@selectdb.com>
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@Jibing-Li Jibing-Li Jibing-Li approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @Thearas @doris-robot @CalvinKirs @Jibing-Li @morrySnow