Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](local exchange) Fix local exchange blocked by a huge data block #38657

Merged
merged 1 commit into from
Aug 1, 2024
Merged

[fix](local exchange) Fix local exchange blocked by a huge data block #38657

merged 1 commit into from
Aug 1, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Aug 1, 2024 •
edited
Loading

Proposed changes

If a huge block is push into local exchanger, it will be blocked due to concurrent problems. This PR use a unique lock to resolve it .

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the doing label Aug 1, 2024
@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@yiguolei yiguolei added not-merge/2.0 do not merge into 2.0 branch p0_b labels Aug 1, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 41575 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9fdf84cea37449872a526d3cd27bfbdb3fde39c, data reload: false

------ Round 1 ----------------------------------
q1	17718	4165	4189	4165
q2	2044	201	195	195
q3	10446	1259	1350	1259
q4	10171	796	946	796
q5	7623	3036	2990	2990
q6	223	138	135	135
q7	1053	614	612	612
q8	9443	1770	1935	1770
q9	8573	6606	6589	6589
q10	8767	3863	3848	3848
q11	437	247	252	247
q12	425	237	233	233
q13	17772	2964	2966	2964
q14	272	241	248	241
q15	520	495	491	491
q16	540	382	382	382
q17	982	921	925	921
q18	7968	7403	7200	7200
q19	1738	1226	1214	1214
q20	553	333	347	333
q21	5413	4713	4747	4713
q22	355	280	277	277
Total cold run time: 113036 ms
Total hot run time: 41575 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4085	4011	4107	4011
q2	333	227	223	223
q3	3025	3033	3144	3033
q4	1995	2039	1986	1986
q5	5573	5507	5482	5482
q6	221	131	131	131
q7	2116	1760	1824	1760
q8	3313	3361	3352	3352
q9	8743	8718	8756	8718
q10	3972	4019	3966	3966
q11	549	446	448	446
q12	769	652	618	618
q13	16413	3103	3120	3103
q14	317	272	276	272
q15	542	487	486	486
q16	463	432	425	425
q17	1748	1733	1752	1733
q18	8219	7828	7691	7691
q19	1763	1718	1711	1711
q20	2041	1894	1840	1840
q21	5809	5457	5488	5457
q22	532	465	489	465
Total cold run time: 72541 ms
Total hot run time: 56909 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170510 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9fdf84cea37449872a526d3cd27bfbdb3fde39c, data reload: false

query1	915	366	364	364
query2	6453	1702	1749	1702
query3	6665	212	235	212
query4	19967	17467	17393	17393
query5	3636	497	499	497
query6	279	162	159	159
query7	4577	289	297	289
query8	249	191	197	191
query9	8531	2403	2391	2391
query10	463	283	265	265
query11	10477	10005	9950	9950
query12	121	94	85	85
query13	1626	379	405	379
query14	9161	8444	8085	8085
query15	197	159	175	159
query16	6836	459	465	459
query17	919	544	535	535
query18	1895	274	269	269
query19	201	144	137	137
query20	86	86	83	83
query21	205	98	98	98
query22	4169	4050	3920	3920
query23	34027	33358	33546	33358
query24	10292	3103	3072	3072
query25	700	438	404	404
query26	1733	148	152	148
query27	2952	288	279	279
query28	7473	2053	2030	2030
query29	1302	437	414	414
query30	234	154	156	154
query31	942	747	756	747
query32	102	58	56	56
query33	679	336	323	323
query34	922	495	528	495
query35	875	792	754	754
query36	1057	889	886	886
query37	300	78	80	78
query38	2945	2887	2759	2759
query39	880	836	819	819
query40	258	117	119	117
query41	49	46	48	46
query42	127	96	100	96
query43	470	438	443	438
query44	1181	736	733	733
query45	215	182	184	182
query46	1086	828	827	827
query47	1800	1701	1698	1698
query48	357	286	301	286
query49	926	426	405	405
query50	908	432	435	432
query51	6695	6682	6681	6681
query52	104	88	87	87
query53	255	180	180	180
query54	621	454	453	453
query55	74	73	76	73
query56	270	250	263	250
query57	1137	1065	1089	1065
query58	270	252	266	252
query59	2568	2461	2447	2447
query60	287	266	272	266
query61	92	91	91	91
query62	886	670	655	655
query63	214	183	179	179
query64	5683	1891	1848	1848
query65	3149	3069	3085	3069
query66	1315	329	333	329
query67	15231	15064	14756	14756
query68	4268	558	562	558
query69	440	291	296	291
query70	1131	1084	1089	1084
query71	347	270	266	266
query72	7069	2663	2469	2469
query73	775	332	332	332
query74	5985	5640	5633	5633
query75	3318	2755	2722	2722
query76	2144	1358	1390	1358
query77	413	310	365	310
query78	9434	8779	8885	8779
query79	1557	537	537	537
query80	1119	487	497	487
query81	549	224	223	223
query82	1061	131	125	125
query83	241	172	164	164
query84	270	78	77	77
query85	1207	321	295	295
query86	395	276	286	276
query87	3287	3108	3100	3100
query88	2973	2503	2494	2494
query89	372	289	289	289
query90	1679	190	196	190
query91	125	101	99	99
query92	58	48	48	48
query93	1387	669	619	619
query94	760	287	289	287
query95	367	262	266	262
query96	592	279	284	279
query97	3236	3081	3045	3045
query98	210	197	196	196
query99	1681	1306	1277	1277
Total cold run time: 259721 ms
Total hot run time: 170510 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9fdf84cea37449872a526d3cd27bfbdb3fde39c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.06	0.06
query5	0.48	0.49	0.48
query6	1.16	0.73	0.71
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.58	0.50	0.52
query10	0.56	0.57	0.56
query11	0.16	0.12	0.11
query12	0.15	0.13	0.13
query13	0.61	0.60	0.60
query14	0.77	0.78	0.79
query15	0.90	0.86	0.85
query16	0.36	0.35	0.35
query17	0.97	1.00	0.98
query18	0.22	0.21	0.21
query19	1.81	1.75	1.72
query20	0.02	0.01	0.01
query21	15.40	0.78	0.66
query22	3.80	7.91	1.37
query23	17.97	1.24	1.19
query24	2.25	0.23	0.22
query25	0.19	0.09	0.08
query26	0.32	0.21	0.22
query27	0.46	0.24	0.23
query28	13.18	1.00	0.97
query29	12.57	3.36	3.35
query30	0.25	0.06	0.05
query31	2.86	0.41	0.40
query32	3.24	0.48	0.48
query33	2.98	2.94	2.97
query34	15.40	4.28	4.30
query35	4.32	4.32	4.31
query36	0.68	0.48	0.48
query37	0.19	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 107.48 s
Total hot run time: 30.04 s

@Gabriel39 Gabriel39 merged commit bb1a9c9 into apache:master Aug 1, 2024
31 of 34 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Aug 1, 2024
@Gabriel39
[fix](local exchange) Fix local exchange blocked by a huge data block (…
e66bd69 
…apache#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
@Gabriel39 Gabriel39 mentioned this pull request Aug 1, 2024
Merged
Gabriel39 added a commit that referenced this pull request Aug 1, 2024
@Gabriel39
[fix](local exchange) Fix local exchange blocked by a huge data block… (
82c6815 
#38693)

… (#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
dataroaring pushed a commit that referenced this pull request Aug 2, 2024
@Gabriel39 @dataroaring
[fix](local exchange) Fix local exchange blocked by a huge data block (…
1814729 
…#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Aug 9, 2024
@Gabriel39 @feiniaofeiafei
[fix](local exchange) Fix local exchange blocked by a huge data block (…
64146ae 
…apache#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
morningman pushed a commit to morningman/doris that referenced this pull request Aug 18, 2024
@Gabriel39 @morningman
[fix](local exchange) Fix local exchange blocked by a huge data block… (
164ba7e 
apache#38693)

… (apache#38657)

If a huge block is push into local exchanger, it will be blocked due to
concurrent problems. This PR use a unique lock to resolve it .
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yiguolei yiguolei yiguolei approved these changes

@BiteTheDDDDt BiteTheDDDDt BiteTheDDDDt approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged doing not-merge/2.0 do not merge into 2.0 branch p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @BiteTheDDDDt @yiguolei @dataroaring