Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](array_range) fix array_range func for large param which should return error #38284

Merged
merged 1 commit into from
Jul 27, 2024
Merged

[fix](array_range) fix array_range func for large param which should return error #38284

merged 1 commit into from
Jul 27, 2024

Conversation

amorynan
Copy link
Contributor

@amorynan amorynan commented Jul 24, 2024 •
edited
Loading

Proposed changes

if array_range with large size , which will make be oom , so we should avoid this with max_array_size_as_field
Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@amorynan
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40270 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 24650e94dffdd733070e139669dc5a3d76162e9d, data reload: false

------ Round 1 ----------------------------------
q1	17902	4353	4287	4287
q2	2011	190	188	188
q3	10470	1206	1172	1172
q4	10200	818	845	818
q5	8793	2706	2737	2706
q6	227	141	142	141
q7	968	633	625	625
q8	9774	2082	2145	2082
q9	9255	6576	6561	6561
q10	8651	3797	3764	3764
q11	452	242	241	241
q12	395	233	229	229
q13	18891	2990	2970	2970
q14	281	235	238	235
q15	517	471	488	471
q16	526	393	378	378
q17	963	695	679	679
q18	8140	7568	7526	7526
q19	6700	1374	1363	1363
q20	689	319	342	319
q21	4859	3227	3258	3227
q22	356	288	291	288
Total cold run time: 121020 ms
Total hot run time: 40270 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4288	4243	4243
q2	373	261	278	261
q3	3022	2784	2738	2738
q4	1859	1653	1680	1653
q5	5315	5334	5315	5315
q6	216	132	135	132
q7	2065	1747	1685	1685
q8	3199	3328	3299	3299
q9	8433	8402	8355	8355
q10	3954	3669	3665	3665
q11	598	496	492	492
q12	803	619	625	619
q13	17596	2997	2989	2989
q14	313	268	290	268
q15	515	482	477	477
q16	513	450	428	428
q17	1774	1490	1453	1453
q18	7707	7506	7378	7378
q19	1658	1380	1650	1380
q20	1990	1773	1781	1773
q21	4815	4698	4748	4698
q22	568	504	498	498
Total cold run time: 71675 ms
Total hot run time: 53799 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174143 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 24650e94dffdd733070e139669dc5a3d76162e9d, data reload: false

query1	905	366	371	366
query2	6455	1893	1829	1829
query3	6646	213	213	213
query4	28547	17579	17507	17507
query5	4226	495	504	495
query6	288	188	165	165
query7	4589	291	283	283
query8	247	204	194	194
query9	8796	2387	2370	2370
query10	446	286	273	273
query11	10674	10091	10242	10091
query12	135	84	83	83
query13	1653	385	373	373
query14	10496	8137	7654	7654
query15	226	172	171	171
query16	7825	516	480	480
query17	1585	591	565	565
query18	1936	284	282	282
query19	217	155	153	153
query20	97	86	86	86
query21	213	129	125	125
query22	4350	4048	4065	4048
query23	33679	32880	33241	32880
query24	11715	2889	2899	2889
query25	651	383	382	382
query26	1520	150	150	150
query27	2988	277	283	277
query28	7472	2023	2006	2006
query29	1024	637	629	629
query30	287	157	151	151
query31	963	745	741	741
query32	97	57	57	57
query33	789	345	344	344
query34	883	480	487	480
query35	875	716	725	716
query36	1093	926	965	926
query37	138	79	81	79
query38	2831	2805	2827	2805
query39	841	813	847	813
query40	280	123	123	123
query41	49	47	47	47
query42	124	97	106	97
query43	500	463	471	463
query44	1175	728	734	728
query45	195	164	163	163
query46	1121	724	731	724
query47	1855	1786	1770	1770
query48	376	299	293	293
query49	1183	421	425	421
query50	799	392	391	391
query51	6936	6790	6605	6605
query52	112	93	100	93
query53	361	286	296	286
query54	930	454	465	454
query55	79	76	76	76
query56	311	288	288	288
query57	1210	1024	1017	1017
query58	267	259	259	259
query59	2870	2644	2650	2644
query60	317	279	283	279
query61	116	97	121	97
query62	825	624	655	624
query63	328	295	288	288
query64	10546	2234	2699	2234
query65	3218	3104	3096	3096
query66	1386	365	342	342
query67	15832	14956	15088	14956
query68	6834	559	553	553
query69	726	461	388	388
query70	1167	1135	1161	1135
query71	547	277	273	273
query72	8340	5621	5914	5621
query73	800	339	325	325
query74	6362	5682	5762	5682
query75	5004	2664	2660	2660
query76	4827	941	927	927
query77	778	316	307	307
query78	9785	9037	9186	9037
query79	10356	526	521	521
query80	1055	501	475	475
query81	579	226	224	224
query82	787	137	132	132
query83	340	166	217	166
query84	275	85	84	84
query85	1376	327	315	315
query86	403	304	310	304
query87	3326	3091	3117	3091
query88	4899	2358	2366	2358
query89	540	373	373	373
query90	2054	193	193	193
query91	131	100	101	100
query92	60	48	51	48
query93	6979	529	525	525
query94	1284	300	290	290
query95	411	321	327	321
query96	628	272	269	269
query97	3177	3011	3060	3011
query98	221	192	196	192
query99	1539	1254	1291	1254
Total cold run time: 307289 ms
Total hot run time: 174143 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 24650e94dffdd733070e139669dc5a3d76162e9d, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.03
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.50	0.48	0.49
query6	1.13	0.73	0.74
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.55	0.48	0.50
query10	0.55	0.56	0.54
query11	0.14	0.12	0.12
query12	0.14	0.12	0.13
query13	0.60	0.59	0.58
query14	0.75	0.78	0.78
query15	0.86	0.84	0.82
query16	0.37	0.36	0.37
query17	1.00	1.02	1.03
query18	0.22	0.21	0.22
query19	1.78	1.70	1.72
query20	0.01	0.01	0.02
query21	15.43	0.77	0.67
query22	4.38	7.10	2.12
query23	18.24	1.45	1.30
query24	1.58	0.39	0.25
query25	0.16	0.09	0.09
query26	0.30	0.20	0.20
query27	0.45	0.23	0.23
query28	13.20	1.03	1.00
query29	12.64	3.34	3.34
query30	0.26	0.06	0.06
query31	2.86	0.40	0.40
query32	3.25	0.48	0.49
query33	2.86	2.97	2.85
query34	17.11	4.40	4.36
query35	4.41	4.43	4.40
query36	0.65	0.47	0.46
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.21 s
Total hot run time: 31 s

@amorynan amorynan changed the title 'improve array_range func for large param [fix](array_range) fix array_range func for large param which should return error Jul 24, 2024
eldenmoon
eldenmoon approved these changes Jul 25, 2024
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

zclllyybb
zclllyybb approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@zclllyybb zclllyybb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@amorynan
Copy link
Contributor Author

run beut

@eldenmoon eldenmoon merged commit 121c200 into apache:master Jul 27, 2024
32 of 35 checks passed
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
@amorynan @dataroaring
[fix](array_range) fix array_range func for large param which should …
058c55e 
…return error (#38284)

if array_range with large size , which will make be oom , so we should
avoid this with max_array_size_as_field
This was referenced Aug 1, 2024
Merged
Merged
xiaokang pushed a commit that referenced this pull request Aug 1, 2024
@amorynan
[fix](array_range) fix array_range func for large param which should …
35651eb 
…return error #38284 (#38712)
yiguolei pushed a commit that referenced this pull request Aug 2, 2024
@amorynan
[pick](array-range)improve array_range func for large param (#38707)
f5bc659 
## Proposed changes
backport: #38284
Issue Number: close #xxx

<!--Describe your changes.-->
@xiaokang xiaokang removed the doing label Aug 6, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@amorynan @Baymine
[fix](array_range) fix array_range func for large param which should …
0a3e437 
…return error apache#38284 (apache#38712)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@eldenmoon eldenmoon eldenmoon approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

@xiaokang xiaokang xiaokang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@amorynan @doris-robot @xiaokang @zclllyybb @eldenmoon @yiguolei @dataroaring