Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feat](nereids) adjust min/max for partition key #41729

Merged
merged 1 commit into from
Oct 21, 2024
Merged

[feat](nereids) adjust min/max for partition key #41729

merged 1 commit into from
Oct 21, 2024

Conversation

englefly
Copy link
Contributor

Proposed changes

after partition prune, adjust the min/max of partition key
Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40970 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

------ Round 1 ----------------------------------
q1	17581	7465	7462	7462
q2	2012	276	275	275
q3	11990	1050	1105	1050
q4	10559	725	789	725
q5	7745	2916	2783	2783
q6	240	156	148	148
q7	983	614	609	609
q8	9350	2005	2000	2000
q9	7008	6397	6397	6397
q10	6991	2304	2328	2304
q11	436	246	248	246
q12	410	218	220	218
q13	17790	2963	2960	2960
q14	243	209	206	206
q15	574	524	533	524
q16	647	583	560	560
q17	978	533	607	533
q18	7335	6702	6625	6625
q19	1355	899	1009	899
q20	477	214	201	201
q21	3997	3269	3299	3269
q22	1127	976	986	976
Total cold run time: 109828 ms
Total hot run time: 40970 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7306	7286	7259	7259
q2	340	236	233	233
q3	3073	2971	2936	2936
q4	2084	1874	1851	1851
q5	5755	5752	5776	5752
q6	232	150	143	143
q7	2321	1834	1821	1821
q8	3389	3550	3460	3460
q9	8859	8840	8806	8806
q10	3604	3544	3569	3544
q11	609	494	484	484
q12	850	633	623	623
q13	9176	3198	3167	3167
q14	330	288	287	287
q15	578	539	519	519
q16	693	651	633	633
q17	1856	1623	1601	1601
q18	8270	7881	7504	7504
q19	1689	1511	1447	1447
q20	2124	1874	1865	1865
q21	5616	5428	5408	5408
q22	1155	1103	1039	1039
Total cold run time: 69909 ms
Total hot run time: 60382 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191854 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

query1	892	395	404	395
query2	6259	2070	2029	2029
query3	8685	199	204	199
query4	34088	23633	23608	23608
query5	3489	487	459	459
query6	277	165	188	165
query7	4194	312	301	301
query8	281	237	218	218
query9	9237	2687	2701	2687
query10	469	300	276	276
query11	17864	15522	15114	15114
query12	154	98	97	97
query13	1574	485	471	471
query14	9735	7651	7303	7303
query15	254	174	177	174
query16	7369	450	471	450
query17	1638	656	601	601
query18	2039	315	335	315
query19	378	159	153	153
query20	124	114	114	114
query21	217	111	109	109
query22	4787	4606	4574	4574
query23	35305	34068	34029	34029
query24	11001	2831	2881	2831
query25	640	411	405	405
query26	1035	161	165	161
query27	2218	309	307	307
query28	7475	2416	2407	2407
query29	758	434	449	434
query30	256	153	148	148
query31	1041	792	799	792
query32	97	54	56	54
query33	763	308	291	291
query34	906	519	520	519
query35	842	727	725	725
query36	1128	952	985	952
query37	147	91	89	89
query38	3984	3958	3923	3923
query39	1480	1441	1408	1408
query40	207	97	98	97
query41	49	45	43	43
query42	120	96	99	96
query43	534	487	485	485
query44	1280	810	806	806
query45	199	167	163	163
query46	1159	696	712	696
query47	1919	1825	1831	1825
query48	447	365	338	338
query49	925	408	407	407
query50	798	419	418	418
query51	7126	6936	6922	6922
query52	96	86	94	86
query53	257	184	182	182
query54	1236	494	486	486
query55	84	77	81	77
query56	304	269	265	265
query57	1265	1149	1154	1149
query58	227	226	238	226
query59	3051	2956	2837	2837
query60	291	263	270	263
query61	108	102	105	102
query62	839	673	659	659
query63	220	188	187	187
query64	4019	639	618	618
query65	3258	3204	3179	3179
query66	730	300	298	298
query67	15750	15405	15639	15405
query68	4718	560	590	560
query69	529	304	295	295
query70	1171	1065	1074	1065
query71	361	277	304	277
query72	7052	3968	3923	3923
query73	759	343	353	343
query74	10131	9005	8918	8918
query75	3429	2679	2634	2634
query76	3520	958	837	837
query77	406	298	298	298
query78	10461	9611	9617	9611
query79	2435	596	612	596
query80	1436	477	477	477
query81	579	239	248	239
query82	702	143	139	139
query83	233	135	134	134
query84	241	78	77	77
query85	1272	297	298	297
query86	385	307	299	299
query87	4375	4290	4356	4290
query88	3756	2478	2417	2417
query89	400	282	294	282
query90	1956	188	186	186
query91	143	105	108	105
query92	61	48	47	47
query93	1643	557	547	547
query94	894	295	294	294
query95	352	256	258	256
query96	617	282	289	282
query97	3238	3111	3052	3052
query98	221	192	189	189
query99	1773	1300	1286	1286
Total cold run time: 298827 ms
Total hot run time: 191854 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

query1	0.04	0.04	0.02
query2	0.06	0.03	0.03
query3	0.23	0.07	0.06
query4	1.64	0.11	0.11
query5	0.53	0.49	0.50
query6	1.14	0.74	0.72
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.51	0.50
query10	0.56	0.55	0.57
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.63	0.60	0.61
query14	2.73	2.72	2.70
query15	0.90	0.84	0.85
query16	0.38	0.39	0.40
query17	1.02	1.05	1.01
query18	0.21	0.20	0.20
query19	1.97	1.88	2.01
query20	0.01	0.01	0.01
query21	15.36	0.60	0.59
query22	2.91	2.87	2.11
query23	17.31	0.98	0.80
query24	2.83	1.27	1.15
query25	0.35	0.24	0.05
query26	0.36	0.15	0.14
query27	0.04	0.04	0.04
query28	10.48	1.09	1.08
query29	12.56	3.28	3.24
query30	0.25	0.06	0.06
query31	2.89	0.39	0.37
query32	3.28	0.46	0.46
query33	3.02	3.02	3.04
query34	17.01	4.46	4.43
query35	4.56	4.50	4.52
query36	0.66	0.50	0.47
query37	0.08	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.03
query40	0.16	0.12	0.12
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.03
Total cold run time: 107.25 s
Total hot run time: 33.17 s

@englefly
Copy link
Contributor Author

run p0

@englefly
Copy link
Contributor Author

run cloud_p0

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41215 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

------ Round 1 ----------------------------------
q1	17585	7344	7270	7270
q2	2036	293	274	274
q3	12158	1084	1131	1084
q4	10561	900	817	817
q5	7757	3130	3080	3080
q6	236	147	148	147
q7	1016	620	591	591
q8	9352	1925	1887	1887
q9	7268	6504	6408	6408
q10	7039	2395	2412	2395
q11	448	236	247	236
q12	411	224	219	219
q13	17766	3029	3035	3029
q14	252	203	217	203
q15	587	533	518	518
q16	666	573	561	561
q17	961	568	565	565
q18	7272	6584	6674	6584
q19	1355	1081	1040	1040
q20	470	187	185	185
q21	3996	3183	3133	3133
q22	1092	989	1021	989
Total cold run time: 110284 ms
Total hot run time: 41215 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7276	7247	7236	7236
q2	325	238	234	234
q3	3032	2920	2965	2920
q4	2119	1792	1836	1792
q5	5773	5747	5824	5747
q6	234	140	143	140
q7	2257	1793	1794	1793
q8	3349	3560	3434	3434
q9	8931	8891	8873	8873
q10	3573	3564	3576	3564
q11	607	485	485	485
q12	863	656	597	597
q13	10609	3161	3211	3161
q14	313	275	268	268
q15	576	532	521	521
q16	697	640	625	625
q17	1859	1609	1611	1609
q18	8350	7895	7645	7645
q19	1705	1405	1516	1405
q20	2140	1883	1870	1870
q21	5546	5584	5542	5542
q22	1162	1068	1063	1063
Total cold run time: 71296 ms
Total hot run time: 60524 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192598 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

query1	908	416	399	399
query2	6235	2042	2019	2019
query3	8689	197	208	197
query4	34417	23675	23620	23620
query5	3505	487	481	481
query6	285	165	165	165
query7	4188	282	283	282
query8	285	229	233	229
query9	9558	2755	2754	2754
query10	453	310	272	272
query11	17991	15360	15390	15360
query12	154	99	101	99
query13	1563	434	410	410
query14	9415	7592	7450	7450
query15	265	174	178	174
query16	7980	499	483	483
query17	1675	608	571	571
query18	2052	324	315	315
query19	372	160	166	160
query20	130	111	113	111
query21	213	109	108	108
query22	5004	4633	4731	4633
query23	35478	34158	35349	34158
query24	11351	2740	2791	2740
query25	530	405	410	405
query26	725	160	161	160
query27	2181	276	281	276
query28	7332	2460	2439	2439
query29	677	423	412	412
query30	260	157	159	157
query31	1032	788	817	788
query32	100	56	54	54
query33	748	293	295	293
query34	906	501	523	501
query35	896	733	758	733
query36	1094	928	946	928
query37	152	88	90	88
query38	3986	3898	3885	3885
query39	1499	1458	1451	1451
query40	206	98	98	98
query41	47	44	46	44
query42	119	102	99	99
query43	520	472	474	472
query44	1245	802	800	800
query45	199	168	168	168
query46	1111	701	719	701
query47	1968	1876	1835	1835
query48	426	321	319	319
query49	926	434	469	434
query50	826	391	371	371
query51	7203	6961	7031	6961
query52	107	91	92	91
query53	263	178	175	175
query54	1211	438	440	438
query55	78	75	76	75
query56	284	277	291	277
query57	1302	1146	1120	1120
query58	228	224	274	224
query59	3109	3099	3021	3021
query60	298	259	259	259
query61	109	101	102	101
query62	840	691	677	677
query63	206	184	178	178
query64	3840	641	615	615
query65	3262	3220	3198	3198
query66	817	296	311	296
query67	16017	15768	15952	15768
query68	4428	559	559	559
query69	551	290	275	275
query70	1188	1069	1098	1069
query71	388	263	306	263
query72	7338	4005	3933	3933
query73	771	353	354	353
query74	10162	8980	9068	8980
query75	4067	2620	2658	2620
query76	3565	919	861	861
query77	625	294	275	275
query78	10485	9768	9641	9641
query79	1614	574	584	574
query80	2467	451	458	451
query81	584	237	239	237
query82	718	138	139	138
query83	300	132	133	132
query84	283	74	74	74
query85	1631	295	286	286
query86	438	308	280	280
query87	4592	4304	4317	4304
query88	3192	2173	2147	2147
query89	402	291	280	280
query90	2086	181	181	181
query91	137	99	99	99
query92	63	47	48	47
query93	1068	518	532	518
query94	1182	292	298	292
query95	354	243	244	243
query96	613	275	286	275
query97	3300	3128	3162	3128
query98	218	195	185	185
query99	1594	1326	1323	1323
Total cold run time: 301455 ms
Total hot run time: 192598 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39a0147b92ea2947a12fbf3a5ac95fa325f43a56, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.50	0.50	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.03	0.04
query9	0.57	0.51	0.49
query10	0.56	0.54	0.53
query11	0.14	0.11	0.11
query12	0.13	0.11	0.11
query13	0.61	0.59	0.58
query14	2.71	2.80	2.77
query15	0.91	0.83	0.84
query16	0.38	0.37	0.38
query17	0.95	1.06	1.03
query18	0.20	0.19	0.20
query19	1.90	1.87	2.04
query20	0.01	0.01	0.01
query21	15.37	0.59	0.57
query22	2.43	2.04	1.70
query23	17.05	0.97	0.79
query24	2.84	1.62	1.18
query25	0.34	0.11	0.20
query26	0.41	0.13	0.13
query27	0.04	0.04	0.04
query28	10.24	1.08	1.06
query29	12.56	3.25	3.23
query30	0.24	0.06	0.06
query31	2.88	0.38	0.36
query32	3.28	0.46	0.46
query33	3.03	2.99	3.04
query34	16.94	4.45	4.43
query35	4.47	4.51	4.54
query36	0.68	0.47	0.49
query37	0.08	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.01 s
Total hot run time: 32.77 s

@englefly
Copy link
Contributor Author

run p0

@englefly
Copy link
Contributor Author

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Oct 21, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly englefly merged commit 948861b into apache:master Oct 21, 2024
26 of 28 checks passed
@englefly englefly deleted the partitionkey-minmax branch November 30, 2024 04:51
englefly added a commit that referenced this pull request Dec 9, 2024
@englefly
[feat](nereids) adjust min/max for partition key #41729 branch-3.0 (#…
90ffe90 
…45110)

### What problem does this PR solve?
pick 41729
@englefly englefly mentioned this pull request Dec 20, 2024
Merged
16 tasks
englefly added a commit that referenced this pull request Dec 22, 2024
@englefly
[feat](nereids) adjust min/max for partition key #41729 branch-2.1 (#…
7d9a751 
…45713)

### What problem does this PR solve?
pick #41729
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Jibing-Li Jibing-Li Jibing-Li left review comments

@starocean999 starocean999 starocean999 approved these changes

@LiBinfeng-01 LiBinfeng-01 LiBinfeng-01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@englefly @doris-robot @starocean999 @LiBinfeng-01 @Jibing-Li @morrySnow