Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](pipeline) Fix mem control in local exchanger #38885

Merged
merged 1 commit into from
Aug 6, 2024
Merged

[fix](pipeline) Fix mem control in local exchanger #38885

merged 1 commit into from
Aug 6, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Aug 5, 2024 •
edited
Loading

Proposed changes

If a block (>128M) is dequeue by local exchange source operator and it is the last block, both of source operators and sink operators will be hang. This PR fixed it.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41661 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 824824d91749ede38d5f84beee2087c72cf264a6, data reload: false

------ Round 1 ----------------------------------
q1	17605	4127	4110	4110
q2	2023	197	204	197
q3	10451	1382	1334	1334
q4	10159	839	929	839
q5	7633	2970	2953	2953
q6	222	138	138	138
q7	1070	609	605	605
q8	9441	1942	1910	1910
q9	8541	6587	6572	6572
q10	8752	3847	3878	3847
q11	436	248	249	248
q12	447	228	221	221
q13	17757	2922	2911	2911
q14	278	243	243	243
q15	527	480	490	480
q16	522	390	383	383
q17	971	902	888	888
q18	8059	7369	7198	7198
q19	1420	1223	1206	1206
q20	565	324	326	324
q21	5278	4777	4791	4777
q22	353	277	280	277
Total cold run time: 112510 ms
Total hot run time: 41661 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4146	4013	4032	4013
q2	331	221	220	220
q3	3042	3016	3149	3016
q4	1982	2006	1940	1940
q5	5614	5488	5449	5449
q6	217	127	128	127
q7	2128	1788	1845	1788
q8	3289	3387	3339	3339
q9	8645	8608	8749	8608
q10	3925	4042	3916	3916
q11	564	466	453	453
q12	794	643	628	628
q13	15915	3093	3144	3093
q14	309	280	282	280
q15	548	501	484	484
q16	452	417	410	410
q17	1801	1766	1748	1748
q18	8242	7850	7672	7672
q19	1709	1703	1720	1703
q20	2089	1854	1833	1833
q21	5715	5514	5259	5259
q22	519	468	453	453
Total cold run time: 71976 ms
Total hot run time: 56432 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170027 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 824824d91749ede38d5f84beee2087c72cf264a6, data reload: false

query1	914	376	372	372
query2	6464	1646	1662	1646
query3	6660	217	221	217
query4	19199	17405	17471	17405
query5	3634	529	506	506
query6	279	166	184	166
query7	4603	294	286	286
query8	267	204	188	188
query9	8510	2385	2359	2359
query10	432	267	271	267
query11	10671	10173	10124	10124
query12	119	89	88	88
query13	1637	395	365	365
query14	9350	6819	6918	6819
query15	211	161	163	161
query16	6808	449	407	407
query17	933	549	553	549
query18	1744	283	273	273
query19	195	138	142	138
query20	109	89	84	84
query21	200	97	97	97
query22	4367	4284	4216	4216
query23	33694	33715	33523	33523
query24	9237	3207	3088	3088
query25	657	398	402	398
query26	1513	152	156	152
query27	2954	280	288	280
query28	7439	2048	1994	1994
query29	1070	414	434	414
query30	230	159	153	153
query31	961	780	789	780
query32	103	58	55	55
query33	671	321	321	321
query34	963	493	505	493
query35	895	789	751	751
query36	1072	874	874	874
query37	168	81	93	81
query38	2920	2794	2774	2774
query39	877	823	820	820
query40	260	109	112	109
query41	46	43	42	42
query42	119	99	95	95
query43	482	424	420	420
query44	1194	729	730	729
query45	210	178	180	178
query46	1080	786	782	782
query47	1832	1751	1778	1751
query48	361	290	286	286
query49	922	426	426	426
query50	897	434	425	425
query51	6791	6730	6701	6701
query52	102	90	90	90
query53	251	180	177	177
query54	599	462	459	459
query55	75	76	77	76
query56	276	253	255	253
query57	1146	1026	1038	1026
query58	269	267	310	267
query59	2509	2330	2309	2309
query60	291	275	273	273
query61	94	96	90	90
query62	900	643	648	643
query63	215	180	182	180
query64	5693	1894	1870	1870
query65	3150	3093	3079	3079
query66	1024	347	337	337
query67	15282	14888	14883	14883
query68	6002	583	591	583
query69	777	409	321	321
query70	1134	1142	1092	1092
query71	499	290	334	290
query72	7668	2676	2517	2517
query73	938	321	323	321
query74	6163	5735	5693	5693
query75	3745	2703	2718	2703
query76	4066	1262	1265	1262
query77	699	309	313	309
query78	9473	9074	8979	8979
query79	1998	543	523	523
query80	1044	511	504	504
query81	545	224	224	224
query82	776	129	129	129
query83	198	171	170	170
query84	266	78	78	78
query85	1358	323	304	304
query86	457	292	290	290
query87	3254	3103	3139	3103
query88	3550	2397	2475	2397
query89	396	302	285	285
query90	1809	190	187	187
query91	124	99	101	99
query92	62	51	51	51
query93	1738	625	614	614
query94	757	291	300	291
query95	381	268	263	263
query96	597	277	280	277
query97	3240	3053	3056	3053
query98	218	203	194	194
query99	1610	1287	1281	1281
Total cold run time: 264342 ms
Total hot run time: 170027 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 824824d91749ede38d5f84beee2087c72cf264a6, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.06	0.08
query5	0.50	0.48	0.48
query6	1.13	0.71	0.71
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.52	0.50
query10	0.58	0.56	0.54
query11	0.15	0.12	0.12
query12	0.16	0.12	0.13
query13	0.61	0.60	0.60
query14	0.77	0.79	0.79
query15	0.91	0.86	0.85
query16	0.35	0.34	0.35
query17	0.99	0.98	0.98
query18	0.22	0.22	0.21
query19	1.83	1.73	1.73
query20	0.02	0.01	0.01
query21	15.39	0.73	0.65
query22	4.09	7.30	1.21
query23	17.83	1.29	1.28
query24	2.32	0.21	0.22
query25	0.18	0.08	0.08
query26	0.31	0.21	0.21
query27	0.47	0.23	0.23
query28	13.15	0.99	0.97
query29	12.56	3.24	3.25
query30	0.26	0.05	0.05
query31	2.88	0.40	0.41
query32	3.23	0.48	0.48
query33	2.92	3.00	2.99
query34	15.44	4.25	4.25
query35	4.28	4.27	4.31
query36	0.67	0.50	0.48
query37	0.19	0.16	0.17
query38	0.17	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 107.58 s
Total hot run time: 29.81 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei removed the doing label Aug 6, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@yiguolei yiguolei merged commit 0ff1ca5 into apache:master Aug 6, 2024
30 of 32 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Aug 6, 2024
@Gabriel39
[fix](pipeline) Fix mem control in local exchanger (apache#38885)
add2865 
## Proposed changes

If a block (>128M) is dequeue by local exchange source operator and it
is the last block, both of source operators and sink operators will be
hang. This PR fixed it.

<!--Describe your changes.-->
@Gabriel39 Gabriel39 mentioned this pull request Aug 6, 2024
Merged
Gabriel39 added a commit that referenced this pull request Aug 6, 2024
@Gabriel39
[fix](pipeline) Fix mem control in local exchanger (#38885) (#38910)
28c0510 
If a block (>128M) is dequeue by local exchange source operator and it
is the last block, both of source operators and sink operators will be
hang. This PR fixed it.

pick #38885
dataroaring pushed a commit that referenced this pull request Aug 6, 2024
@Gabriel39 @dataroaring
[fix](pipeline) Fix mem control in local exchanger (#38885)
33ec829 
## Proposed changes

If a block (>128M) is dequeue by local exchange source operator and it
is the last block, both of source operators and sink operators will be
hang. This PR fixed it.

<!--Describe your changes.-->
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Mryange Mryange Mryange approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@Gabriel39 @doris-robot @yiguolei @Mryange @dataroaring