Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](mtmv) Choose a valid partition column when there are both valid and invalid expressions #38367

Merged
merged 4 commits into from
Jul 31, 2024
Merged

[fix](mtmv) Choose a valid partition column when there are both valid and invalid expressions #38367

merged 4 commits into from
Jul 31, 2024

Conversation

liutang123
Copy link
Contributor

Proposed changes

Choose a partition column when there are both valid and invalid expressions.
Issue Number: close #38366

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liutang123
Copy link
Contributor Author

Will add regression tests later

@liutang123 liutang123 changed the title [fix] (mtmv) Choose a valid partition column when there are both valid and invalid expressions [fix](mtmv) Choose a valid partition column when there are both valid and invalid expressions Jul 25, 2024
@seawinde
Copy link
Contributor

seawinde commented Jul 25, 2024 •
edited
Loading

Could you add some tests for this fix? Partition track test is all in org.apache.doris.nereids.rules.exploration.mv.MaterializedViewUtilsTest

@morrySnow morrySnow added usercase Important user case type label p0_b labels Jul 26, 2024
@xy720
Copy link
Member

xy720 commented Jul 26, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39528 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2ed59caa9985a8f6cd05e027e4e27e611dc76afb, data reload: false

------ Round 1 ----------------------------------
q1	17610	4333	4314	4314
q2	2017	198	196	196
q3	10446	1216	1111	1111
q4	10157	814	783	783
q5	7629	2676	2678	2676
q6	223	137	139	137
q7	962	596	587	587
q8	9221	1925	1897	1897
q9	8733	6551	6554	6551
q10	8915	3765	3794	3765
q11	461	257	249	249
q12	438	232	226	226
q13	18837	2989	3001	2989
q14	290	239	232	232
q15	523	523	495	495
q16	501	411	391	391
q17	971	664	669	664
q18	8023	7460	7390	7390
q19	6710	993	1006	993
q20	678	347	335	335
q21	5101	3267	3261	3261
q22	346	290	286	286
Total cold run time: 118792 ms
Total hot run time: 39528 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4619	4273	4236	4236
q2	366	280	277	277
q3	2984	2892	2921	2892
q4	2066	1707	1767	1707
q5	5608	5534	5476	5476
q6	232	134	134	134
q7	2214	1870	1852	1852
q8	3248	3459	3431	3431
q9	8737	8811	8892	8811
q10	4234	3921	4007	3921
q11	600	510	522	510
q12	845	699	685	685
q13	15978	3175	3194	3175
q14	308	299	298	298
q15	518	491	497	491
q16	501	443	435	435
q17	1812	1533	1519	1519
q18	8122	7929	7780	7780
q19	1728	1558	1561	1558
q20	2177	1892	1895	1892
q21	5102	5076	4834	4834
q22	559	511	497	497
Total cold run time: 72558 ms
Total hot run time: 56411 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173764 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2ed59caa9985a8f6cd05e027e4e27e611dc76afb, data reload: false

query1	895	372	361	361
query2	6433	1947	1861	1861
query3	6642	209	217	209
query4	28570	17543	17266	17266
query5	3689	491	515	491
query6	277	189	171	171
query7	4585	295	290	290
query8	247	208	205	205
query9	8488	2496	2448	2448
query10	447	281	278	278
query11	11461	10113	10004	10004
query12	123	97	86	86
query13	1628	387	368	368
query14	10325	7880	7872	7872
query15	228	167	166	166
query16	7220	480	442	442
query17	1598	568	548	548
query18	1873	284	275	275
query19	192	148	143	143
query20	92	86	86	86
query21	217	100	103	100
query22	4285	3892	4006	3892
query23	34128	33880	33791	33791
query24	8925	2958	2897	2897
query25	610	403	386	386
query26	725	180	155	155
query27	2227	277	281	277
query28	6227	2056	2080	2056
query29	754	443	426	426
query30	253	155	155	155
query31	991	778	762	762
query32	94	52	53	52
query33	661	354	342	342
query34	884	493	504	493
query35	879	748	759	748
query36	1109	962	950	950
query37	139	83	83	83
query38	2970	2910	2763	2763
query39	902	880	841	841
query40	196	120	110	110
query41	47	45	43	43
query42	115	95	103	95
query43	509	466	452	452
query44	1076	729	723	723
query45	210	178	181	178
query46	1090	741	761	741
query47	1816	1741	1762	1741
query48	368	289	298	289
query49	841	422	415	415
query50	791	406	416	406
query51	6953	6645	6707	6645
query52	102	95	93	93
query53	259	182	181	181
query54	877	447	463	447
query55	77	75	72	72
query56	299	300	296	296
query57	1114	1050	1053	1050
query58	261	258	272	258
query59	2984	2759	2635	2635
query60	329	287	291	287
query61	104	97	95	95
query62	772	646	656	646
query63	208	180	182	180
query64	9172	2280	1714	1714
query65	3175	3103	3108	3103
query66	749	333	339	333
query67	15440	14721	14886	14721
query68	4495	536	545	536
query69	606	404	326	326
query70	1183	1099	1145	1099
query71	395	279	273	273
query72	7648	5789	5569	5569
query73	752	329	324	324
query74	6098	5718	5705	5705
query75	3347	2702	2665	2665
query76	2589	969	988	969
query77	631	322	316	316
query78	9973	9191	9068	9068
query79	2114	536	531	531
query80	1184	489	487	487
query81	588	217	228	217
query82	767	142	136	136
query83	274	177	177	177
query84	238	82	80	80
query85	1512	377	319	319
query86	469	311	291	291
query87	3212	3100	3094	3094
query88	4277	2506	2463	2463
query89	390	279	284	279
query90	1754	204	198	198
query91	129	106	116	106
query92	56	47	51	47
query93	2543	531	527	527
query94	858	311	296	296
query95	363	273	268	268
query96	606	285	279	279
query97	3228	3033	3003	3003
query98	222	202	200	200
query99	1546	1272	1228	1228
Total cold run time: 275974 ms
Total hot run time: 173764 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.8 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2ed59caa9985a8f6cd05e027e4e27e611dc76afb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.07
query5	0.51	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.49	0.48
query10	0.54	0.55	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.59	0.58
query14	0.77	0.79	0.78
query15	0.85	0.81	0.82
query16	0.37	0.36	0.36
query17	1.00	0.97	0.97
query18	0.23	0.22	0.22
query19	1.91	1.71	1.83
query20	0.01	0.00	0.01
query21	15.39	0.79	0.65
query22	4.65	7.19	2.03
query23	18.29	1.44	1.34
query24	2.16	0.23	0.22
query25	0.16	0.09	0.09
query26	0.31	0.22	0.21
query27	0.46	0.23	0.23
query28	13.23	1.03	1.00
query29	12.61	3.37	3.37
query30	0.25	0.06	0.05
query31	2.86	0.38	0.40
query32	3.24	0.49	0.47
query33	2.92	2.91	2.89
query34	17.08	4.32	4.34
query35	4.39	4.35	4.42
query36	0.65	0.47	0.47
query37	0.19	0.15	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.04
query40	0.15	0.13	0.13
query41	0.10	0.04	0.04
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 110.26 s
Total hot run time: 30.8 s

@liutang123
Copy link
Contributor Author

run buildall

xy720
xy720 previously approved these changes Jul 29, 2024
View reviewed changes
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 39849 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 59e096591d015c6a0fb1d4a5c8b6150426d89f5c, data reload: false

------ Round 1 ----------------------------------
q1	18395	6111	4371	4371
q2	2025	204	198	198
q3	10563	1243	1133	1133
q4	10303	774	761	761
q5	7540	2764	2708	2708
q6	223	139	137	137
q7	977	597	598	597
q8	9213	1939	1977	1939
q9	8794	6619	6593	6593
q10	8743	3836	3813	3813
q11	460	259	244	244
q12	409	227	222	222
q13	18799	3000	2997	2997
q14	290	241	256	241
q15	520	488	498	488
q16	488	413	388	388
q17	985	730	674	674
q18	8138	7665	7470	7470
q19	5450	1043	1028	1028
q20	675	350	352	350
q21	4969	3212	3210	3210
q22	348	291	287	287
Total cold run time: 118307 ms
Total hot run time: 39849 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4603	4217	4221	4217
q2	374	277	266	266
q3	3051	2751	2739	2739
q4	1900	1651	1640	1640
q5	5350	5289	5325	5289
q6	220	129	130	129
q7	2091	1673	1750	1673
q8	3205	3374	3312	3312
q9	8513	8469	8460	8460
q10	3885	3695	3716	3695
q11	578	498	489	489
q12	754	644	606	606
q13	16349	3013	2971	2971
q14	301	267	275	267
q15	519	466	485	466
q16	462	420	416	416
q17	1823	1516	1481	1481
q18	7812	7698	7427	7427
q19	1670	1562	1512	1512
q20	1995	1803	1782	1782
q21	5338	4592	4605	4592
q22	586	525	491	491
Total cold run time: 71379 ms
Total hot run time: 53920 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173656 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 59e096591d015c6a0fb1d4a5c8b6150426d89f5c, data reload: false

query1	911	374	368	368
query2	6437	1960	1849	1849
query3	6654	210	216	210
query4	28223	17590	17227	17227
query5	4180	491	478	478
query6	263	176	164	164
query7	4583	303	292	292
query8	254	226	205	205
query9	8578	2495	2451	2451
query10	465	270	261	261
query11	11717	10222	10037	10037
query12	140	93	83	83
query13	1638	373	369	369
query14	10147	8682	8393	8393
query15	228	164	167	164
query16	7709	467	473	467
query17	1587	570	526	526
query18	1884	290	290	290
query19	187	141	142	141
query20	104	91	83	83
query21	209	103	101	101
query22	4215	4046	3957	3957
query23	33584	34137	33229	33229
query24	11103	2888	2865	2865
query25	547	369	371	369
query26	689	146	146	146
query27	2196	272	277	272
query28	6078	2059	2030	2030
query29	635	418	409	409
query30	285	147	149	147
query31	971	742	733	733
query32	94	54	57	54
query33	785	329	325	325
query34	890	472	478	472
query35	869	727	733	727
query36	1110	914	918	914
query37	137	81	84	81
query38	2881	2742	2739	2739
query39	857	801	815	801
query40	191	120	120	120
query41	48	45	46	45
query42	120	94	97	94
query43	512	443	472	443
query44	1242	725	732	725
query45	213	182	177	177
query46	1099	713	735	713
query47	1848	1773	1753	1753
query48	372	301	296	296
query49	1079	418	406	406
query50	801	402	408	402
query51	6843	6714	6588	6588
query52	106	88	93	88
query53	257	193	183	183
query54	906	441	445	441
query55	74	74	79	74
query56	304	279	269	269
query57	1151	1042	1075	1042
query58	270	255	274	255
query59	2898	2825	2702	2702
query60	310	288	289	288
query61	99	95	97	95
query62	841	640	648	640
query63	217	183	189	183
query64	9484	2233	1727	1727
query65	3186	3103	3115	3103
query66	820	341	351	341
query67	15103	15043	14749	14749
query68	4638	554	563	554
query69	448	303	303	303
query70	1102	1112	1123	1112
query71	351	270	270	270
query72	6823	5769	5657	5657
query73	749	331	329	329
query74	6117	5715	5621	5621
query75	3433	2753	2707	2707
query76	2370	1100	1125	1100
query77	492	309	317	309
query78	9693	9083	9119	9083
query79	1989	526	516	516
query80	2013	489	482	482
query81	610	223	220	220
query82	839	142	133	133
query83	308	184	181	181
query84	269	77	82	77
query85	1178	386	307	307
query86	410	307	298	298
query87	3239	3073	3042	3042
query88	3766	2485	2485	2485
query89	396	282	290	282
query90	1741	192	196	192
query91	128	118	104	104
query92	63	49	52	49
query93	1547	529	534	529
query94	881	284	273	273
query95	356	260	263	260
query96	592	281	282	281
query97	3295	3077	3083	3077
query98	218	210	200	200
query99	1515	1278	1252	1252
Total cold run time: 275987 ms
Total hot run time: 173656 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 59e096591d015c6a0fb1d4a5c8b6150426d89f5c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.05
query3	0.23	0.05	0.04
query4	1.67	0.08	0.09
query5	0.54	0.48	0.49
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.50
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.16	0.12	0.13
query13	0.60	0.58	0.59
query14	0.76	0.77	0.79
query15	0.86	0.82	0.81
query16	0.36	0.35	0.37
query17	1.02	1.01	1.04
query18	0.23	0.22	0.22
query19	1.74	1.71	1.83
query20	0.01	0.01	0.01
query21	15.40	0.76	0.65
query22	4.84	6.62	1.97
query23	18.19	1.43	1.37
query24	2.08	0.25	0.21
query25	0.15	0.10	0.08
query26	0.29	0.22	0.21
query27	0.46	0.23	0.23
query28	13.26	1.01	1.00
query29	12.65	3.33	3.30
query30	0.26	0.06	0.06
query31	2.85	0.39	0.38
query32	3.28	0.46	0.48
query33	2.90	2.95	2.95
query34	16.93	4.36	4.36
query35	4.46	4.44	4.41
query36	0.67	0.50	0.49
query37	0.18	0.15	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.16	0.11	0.12
query41	0.10	0.06	0.07
query42	0.05	0.04	0.04
query43	0.05	0.03	0.04
Total cold run time: 110.16 s
Total hot run time: 30.86 s

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41513 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ce8cccedc19365b0209c98b9f06298a17a781897, data reload: false

------ Round 1 ----------------------------------
q1	17616	4077	4007	4007
q2	2024	208	204	204
q3	10457	1218	1321	1218
q4	10179	833	943	833
q5	8652	2997	3002	2997
q6	227	144	135	135
q7	1028	620	635	620
q8	9457	1938	1950	1938
q9	8415	6589	6611	6589
q10	8699	3847	3815	3815
q11	419	256	260	256
q12	402	230	226	226
q13	17745	2923	2942	2923
q14	266	244	244	244
q15	519	477	494	477
q16	491	380	393	380
q17	965	931	904	904
q18	8044	7244	7182	7182
q19	1383	1209	1196	1196
q20	570	327	357	327
q21	5335	4811	4760	4760
q22	357	285	282	282
Total cold run time: 113250 ms
Total hot run time: 41513 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4051	4016	4003	4003
q2	323	219	219	219
q3	2996	2962	2977	2962
q4	1864	1878	1858	1858
q5	5239	5268	5218	5218
q6	215	133	132	132
q7	2071	1663	1686	1663
q8	3223	3288	3288	3288
q9	8288	8273	8253	8253
q10	3767	3851	3852	3851
q11	545	459	443	443
q12	701	544	552	544
q13	14991	2963	2936	2936
q14	290	260	264	260
q15	526	475	476	475
q16	447	395	386	386
q17	1719	1697	1701	1697
q18	7623	7400	7337	7337
q19	1671	1661	1682	1661
q20	1969	1770	1744	1744
q21	5472	5193	5242	5193
q22	519	467	498	467
Total cold run time: 68510 ms
Total hot run time: 54590 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168628 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ce8cccedc19365b0209c98b9f06298a17a781897, data reload: false

query1	910	375	365	365
query2	6491	1619	1621	1619
query3	6665	212	223	212
query4	19600	17512	17307	17307
query5	4316	511	520	511
query6	301	174	172	172
query7	4615	311	295	295
query8	255	200	198	198
query9	8517	2417	2410	2410
query10	449	281	262	262
query11	10720	9989	10230	9989
query12	142	97	89	89
query13	1629	378	378	378
query14	10463	7930	7211	7211
query15	214	166	162	162
query16	7136	446	412	412
query17	946	559	562	559
query18	1917	282	289	282
query19	182	141	142	141
query20	95	88	83	83
query21	205	104	101	101
query22	4063	3870	3948	3870
query23	33772	32924	32826	32826
query24	10336	3092	3010	3010
query25	684	389	385	385
query26	1743	148	150	148
query27	2923	276	282	276
query28	6930	2015	1992	1992
query29	1351	420	428	420
query30	289	147	150	147
query31	908	789	757	757
query32	102	54	54	54
query33	694	301	307	301
query34	912	478	499	478
query35	870	746	731	731
query36	989	856	871	856
query37	296	81	82	81
query38	2968	2795	2779	2779
query39	870	805	794	794
query40	279	113	112	112
query41	50	43	45	43
query42	123	98	105	98
query43	466	412	401	401
query44	1177	722	728	722
query45	202	179	179	179
query46	1091	817	794	794
query47	1788	1704	1739	1704
query48	363	295	294	294
query49	1189	420	418	418
query50	912	445	440	440
query51	6766	6659	6712	6659
query52	100	91	87	87
query53	255	188	176	176
query54	681	465	455	455
query55	79	77	76	76
query56	269	273	265	265
query57	1166	1068	1045	1045
query58	285	289	285	285
query59	2747	2402	2300	2300
query60	291	275	273	273
query61	101	98	100	98
query62	906	677	658	658
query63	221	184	189	184
query64	5888	1938	1887	1887
query65	3124	3100	3107	3100
query66	1451	339	334	334
query67	15022	14769	14590	14590
query68	4343	566	579	566
query69	451	304	305	304
query70	1096	1085	1088	1085
query71	410	294	278	278
query72	7140	2693	2502	2502
query73	776	331	332	331
query74	6046	5578	5622	5578
query75	3385	2777	2740	2740
query76	2534	1330	1428	1330
query77	427	310	306	306
query78	9385	8863	8786	8786
query79	1374	539	547	539
query80	1049	542	526	526
query81	575	225	226	225
query82	1057	135	130	130
query83	252	178	174	174
query84	283	84	86	84
query85	1366	372	390	372
query86	396	318	294	294
query87	3277	3122	3064	3064
query88	3028	2495	2500	2495
query89	392	289	297	289
query90	1796	190	189	189
query91	128	106	102	102
query92	65	53	50	50
query93	1433	633	623	623
query94	868	300	288	288
query95	374	268	273	268
query96	602	285	281	281
query97	3218	3011	3034	3011
query98	218	189	202	189
query99	1659	1289	1294	1289
Total cold run time: 262863 ms
Total hot run time: 168628 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ce8cccedc19365b0209c98b9f06298a17a781897, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.08	0.07
query5	0.50	0.48	0.47
query6	1.14	0.72	0.71
query7	0.02	0.02	0.01
query8	0.06	0.04	0.04
query9	0.56	0.53	0.50
query10	0.55	0.61	0.56
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.61	0.60	0.59
query14	0.77	0.78	0.81
query15	0.91	0.87	0.86
query16	0.36	0.35	0.35
query17	1.00	1.00	1.02
query18	0.22	0.21	0.21
query19	1.85	1.69	1.75
query20	0.01	0.01	0.01
query21	15.42	0.79	0.67
query22	4.08	9.16	1.05
query23	17.80	1.37	1.34
query24	2.26	0.22	0.22
query25	0.18	0.08	0.08
query26	0.32	0.22	0.22
query27	0.46	0.24	0.24
query28	13.16	1.00	0.98
query29	12.59	3.42	3.39
query30	0.26	0.06	0.06
query31	2.87	0.41	0.42
query32	3.22	0.49	0.48
query33	2.94	2.95	2.94
query34	15.43	4.26	4.20
query35	4.29	4.28	4.28
query36	0.68	0.48	0.48
query37	0.19	0.17	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.14
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.04	0.04	0.05
Total cold run time: 107.58 s
Total hot run time: 29.81 s

@liutang123
Copy link
Contributor Author

@seawinde hello, this PR is ready, do you have time to take a look?

@liutang123
Copy link
Contributor Author

run buildall

@seawinde
Copy link
Contributor

LGTM!

@doris-robot
Copy link

TPC-H: Total hot run time: 41595 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b2260f38afec6bfe59d51dc27f1a6e5695eea66, data reload: false

------ Round 1 ----------------------------------
q1	18193	4219	4195	4195
q2	2654	205	217	205
q3	11478	1375	1366	1366
q4	10559	854	928	854
q5	8237	2916	3022	2916
q6	222	140	136	136
q7	1036	605	598	598
q8	9440	1860	1917	1860
q9	8468	6606	6610	6606
q10	8719	3841	3829	3829
q11	426	253	253	253
q12	411	223	229	223
q13	17745	2934	2940	2934
q14	267	242	241	241
q15	525	475	480	475
q16	490	389	382	382
q17	948	907	818	818
q18	8161	7305	7168	7168
q19	1400	1229	1226	1226
q20	575	320	327	320
q21	5351	4705	4782	4705
q22	358	285	286	285
Total cold run time: 115663 ms
Total hot run time: 41595 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4097	4053	4058	4053
q2	340	226	218	218
q3	2993	3007	2979	2979
q4	1914	1887	1889	1887
q5	5251	5234	5212	5212
q6	216	130	128	128
q7	2029	1682	1675	1675
q8	3220	3342	3303	3303
q9	8257	8279	8301	8279
q10	3766	3842	3819	3819
q11	543	451	448	448
q12	701	557	569	557
q13	14760	2973	2960	2960
q14	291	259	256	256
q15	515	474	469	469
q16	438	394	391	391
q17	1724	1702	1694	1694
q18	7840	7426	7171	7171
q19	1665	1702	1672	1672
q20	1995	1745	1766	1745
q21	5551	5183	5255	5183
q22	499	455	448	448
Total cold run time: 68605 ms
Total hot run time: 54547 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169330 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b2260f38afec6bfe59d51dc27f1a6e5695eea66, data reload: false

query1	919	368	370	368
query2	6473	1753	1799	1753
query3	6669	213	228	213
query4	19090	17501	17435	17435
query5	4329	505	501	501
query6	268	188	158	158
query7	4598	298	297	297
query8	259	209	190	190
query9	8497	2361	2374	2361
query10	479	281	255	255
query11	10633	10037	10080	10037
query12	142	87	88	87
query13	1621	373	379	373
query14	9518	7834	7173	7173
query15	196	160	162	160
query16	7085	454	467	454
query17	937	538	537	537
query18	1917	285	286	285
query19	193	143	167	143
query20	89	91	88	88
query21	203	107	100	100
query22	4336	4299	4022	4022
query23	33871	32992	32898	32898
query24	10258	3008	3047	3008
query25	708	400	391	391
query26	1774	149	145	145
query27	3033	267	268	267
query28	7013	1962	1964	1962
query29	1334	415	445	415
query30	294	149	152	149
query31	919	757	736	736
query32	102	53	53	53
query33	704	308	316	308
query34	918	481	479	479
query35	848	724	715	715
query36	1012	849	891	849
query37	208	77	77	77
query38	2850	2779	2753	2753
query39	881	852	810	810
query40	280	111	113	111
query41	47	46	44	44
query42	117	97	99	97
query43	476	422	434	422
query44	1151	717	720	717
query45	205	178	176	176
query46	1085	821	791	791
query47	1814	1739	1709	1709
query48	353	289	287	287
query49	1177	427	421	421
query50	900	438	439	438
query51	6683	6701	6550	6550
query52	109	89	94	89
query53	252	183	174	174
query54	618	445	459	445
query55	80	76	75	75
query56	272	249	256	249
query57	1151	1046	1061	1046
query58	283	266	257	257
query59	2568	2604	2457	2457
query60	297	270	283	270
query61	104	98	104	98
query62	930	674	671	671
query63	210	188	181	181
query64	5913	1915	1995	1915
query65	3151	3088	3124	3088
query66	1436	338	331	331
query67	15297	15098	14988	14988
query68	4386	547	562	547
query69	456	298	297	297
query70	1112	1065	1074	1065
query71	359	267	263	263
query72	7113	2722	2526	2526
query73	764	322	326	322
query74	5985	5650	5615	5615
query75	3355	2760	2715	2715
query76	2317	1355	1415	1355
query77	450	307	303	303
query78	10730	9367	8988	8988
query79	2345	530	525	525
query80	906	509	504	504
query81	560	225	228	225
query82	625	132	127	127
query83	267	171	164	164
query84	272	83	79	79
query85	787	421	298	298
query86	474	300	295	295
query87	3279	3095	3158	3095
query88	3830	2397	2400	2397
query89	380	289	286	286
query90	1949	198	189	189
query91	123	101	103	101
query92	66	52	50	50
query93	1372	605	596	596
query94	869	314	292	292
query95	375	263	264	263
query96	603	278	279	278
query97	3156	3011	3041	3011
query98	230	206	192	192
query99	1651	1312	1265	1265
Total cold run time: 263283 ms
Total hot run time: 169330 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b2260f38afec6bfe59d51dc27f1a6e5695eea66, data reload: false

query1	0.05	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.06	0.06
query5	0.48	0.49	0.49
query6	1.16	0.71	0.71
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.58	0.52	0.51
query10	0.57	0.56	0.57
query11	0.15	0.13	0.12
query12	0.15	0.13	0.12
query13	0.63	0.61	0.61
query14	0.78	0.80	0.79
query15	0.90	0.87	0.87
query16	0.36	0.36	0.36
query17	1.01	1.01	1.00
query18	0.23	0.21	0.21
query19	1.84	1.78	1.81
query20	0.01	0.01	0.01
query21	15.40	0.76	0.66
query22	3.59	6.35	1.44
query23	17.89	1.31	1.27
query24	2.17	0.22	0.22
query25	0.18	0.09	0.08
query26	0.32	0.21	0.22
query27	0.46	0.23	0.22
query28	13.18	1.00	0.98
query29	12.63	3.33	3.31
query30	0.27	0.06	0.05
query31	2.87	0.40	0.40
query32	3.24	0.49	0.48
query33	2.93	2.97	2.93
query34	15.44	4.28	4.27
query35	4.33	4.30	4.29
query36	0.67	0.47	0.47
query37	0.19	0.16	0.16
query38	0.17	0.16	0.15
query39	0.05	0.03	0.03
query40	0.16	0.13	0.13
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.06	0.05	0.03
Total cold run time: 107.28 s
Total hot run time: 30.24 s

@liutang123
Copy link
Contributor Author

run feut

@morrySnow
Copy link
Contributor

run cloud_p0

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 31, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit aea92fe into apache:master Jul 31, 2024
28 of 30 checks passed
liutang123 added a commit to liutang123/doris that referenced this pull request Aug 1, 2024
@liutang123
[fix](mtmv) Choose a valid partition column when there are both valid…
bb81732 
… and invalid expressions (apache#38367)

Choose a partition column when there are both valid and invalid
expressions.
@liutang123 liutang123 mentioned this pull request Aug 1, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 1, 2024
@liutang123
[fix](mtmv) Choose a valid partition column when there are both valid…
2562cf3 
… and invalid expressions (#38367) (#38684)

## Proposed changes
pick #38367 

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Aug 2, 2024
@liutang123 @dataroaring
[fix](mtmv) Choose a valid partition column when there are both valid…
6295f40 
… and invalid expressions (#38367)

Choose a partition column when there are both valid and invalid
expressions.
feiniaofeiafei pushed a commit to feiniaofeiafei/doris that referenced this pull request Aug 9, 2024
@liutang123 @feiniaofeiafei
[fix](mtmv) Choose a valid partition column when there are both valid…
06227a5 
… and invalid expressions (apache#38367)

Choose a partition column when there are both valid and invalid
expressions.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seawinde seawinde seawinde approved these changes

@morrySnow morrySnow morrySnow approved these changes

@xy720 xy720 xy720 left review comments

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged p0_b reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
7 participants
@liutang123 @doris-robot @seawinde @xy720 @morrySnow @yiguolei @dataroaring