Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix](nereids) fix partition_prune or expression evaluate wrongly #38897

Merged
merged 2 commits into from
Aug 6, 2024

Conversation

feiniaofeiafei
Copy link
Contributor

@feiniaofeiafei feiniaofeiafei commented Aug 5, 2024

introduced by #38025
Or expression evaluate wrongly in #38025. For example, partition column is a, b, predicate is a=a or b>1 , partition column range is min<a<5, b is infinite. The evaluate result of or expression should be b is infinite(because when a is not null, a=a is true, there is no restrict for b, e.g. a=1, b=0 satisfy a=a or b>1). #38025 pr has wrongly evaluate result b >1

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

924060929
924060929 previously approved these changes Aug 5, 2024
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Aug 5, 2024
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by anyone and no changes requested.

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41358 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0ae23d8df66084fce8d3c6002bf80101da8946b, data reload: false

------ Round 1 ----------------------------------
q1	17616	4075	4075	4075
q2	2022	212	208	208
q3	10433	1214	1298	1214
q4	10172	767	947	767
q5	7639	2960	2946	2946
q6	221	136	134	134
q7	1052	608	613	608
q8	9433	1886	1915	1886
q9	8529	6574	6591	6574
q10	8733	3870	3813	3813
q11	438	259	256	256
q12	413	225	226	225
q13	17759	2913	2921	2913
q14	274	244	250	244
q15	519	498	499	498
q16	523	398	400	398
q17	950	938	892	892
q18	7999	7284	7318	7284
q19	1811	1194	1204	1194
q20	559	331	334	331
q21	5301	4622	4738	4622
q22	348	284	276	276
Total cold run time: 112744 ms
Total hot run time: 41358 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4067	4038	4028	4028
q2	329	225	216	216
q3	2968	2981	3085	2981
q4	1992	2052	1956	1956
q5	5627	5406	5422	5406
q6	229	129	130	129
q7	2098	1772	1807	1772
q8	3268	3348	3308	3308
q9	8632	8594	8829	8594
q10	3905	4035	3893	3893
q11	553	458	454	454
q12	760	616	621	616
q13	16440	3100	3124	3100
q14	298	271	274	271
q15	531	480	498	480
q16	460	404	397	397
q17	1761	1718	1720	1718
q18	8156	7823	7744	7744
q19	1703	1699	1706	1699
q20	2110	1826	1841	1826
q21	5717	5442	5269	5269
q22	541	471	455	455
Total cold run time: 72145 ms
Total hot run time: 56312 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169246 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0ae23d8df66084fce8d3c6002bf80101da8946b, data reload: false

query1	926	380	359	359
query2	6491	1679	1698	1679
query3	6650	221	223	221
query4	20004	17408	17281	17281
query5	3607	502	500	500
query6	281	158	155	155
query7	4596	295	285	285
query8	254	187	202	187
query9	8491	2343	2360	2343
query10	443	258	269	258
query11	10597	10100	10173	10100
query12	118	86	84	84
query13	1622	366	364	364
query14	8450	7577	6181	6181
query15	206	165	163	163
query16	6793	439	458	439
query17	974	563	535	535
query18	1901	271	280	271
query19	200	158	141	141
query20	92	84	87	84
query21	197	100	106	100
query22	4158	3985	3992	3985
query23	33632	33613	33786	33613
query24	10208	3147	3111	3111
query25	721	430	428	428
query26	1778	152	157	152
query27	3059	277	290	277
query28	7552	2037	2007	2007
query29	1319	457	454	454
query30	232	155	154	154
query31	964	776	785	776
query32	104	65	59	59
query33	691	325	337	325
query34	928	488	496	488
query35	874	757	791	757
query36	1033	887	889	887
query37	211	83	93	83
query38	3011	2865	2932	2865
query39	866	842	819	819
query40	258	115	114	114
query41	48	45	46	45
query42	121	108	105	105
query43	471	415	449	415
query44	1195	740	747	740
query45	220	179	180	179
query46	1087	798	779	779
query47	1821	1692	1738	1692
query48	369	290	290	290
query49	959	440	434	434
query50	915	435	434	434
query51	6859	6741	6701	6701
query52	110	92	100	92
query53	264	184	179	179
query54	626	455	452	452
query55	76	73	78	73
query56	289	268	257	257
query57	1149	1032	1069	1032
query58	271	278	284	278
query59	2657	2379	2638	2379
query60	296	289	279	279
query61	113	113	115	113
query62	889	659	671	659
query63	223	187	184	184
query64	5735	2005	1981	1981
query65	3178	3101	3097	3097
query66	1325	340	340	340
query67	15556	14873	14778	14778
query68	6637	572	700	572
query69	718	408	324	324
query70	1115	1036	1046	1036
query71	533	280	277	277
query72	8192	2653	2527	2527
query73	958	321	325	321
query74	6105	5668	5628	5628
query75	4309	2729	2727	2727
query76	4350	1267	1302	1267
query77	809	321	315	315
query78	9643	8966	8890	8890
query79	4793	533	533	533
query80	2039	501	504	501
query81	560	227	228	227
query82	801	129	131	129
query83	283	180	169	169
query84	265	80	118	80
query85	1473	302	304	302
query86	448	314	290	290
query87	3308	3115	3086	3086
query88	4216	2392	2432	2392
query89	468	296	292	292
query90	2006	190	189	189
query91	123	98	101	98
query92	64	50	49	49
query93	5415	621	627	621
query94	951	292	282	282
query95	414	259	265	259
query96	627	279	281	279
query97	3242	3031	3059	3031
query98	242	191	187	187
query99	1606	1299	1295	1295
Total cold run time: 278027 ms
Total hot run time: 169246 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.64 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0ae23d8df66084fce8d3c6002bf80101da8946b, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.49
query6	1.16	0.71	0.71
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.57	0.51	0.54
query10	0.56	0.56	0.56
query11	0.15	0.12	0.12
query12	0.16	0.13	0.13
query13	0.62	0.62	0.60
query14	0.78	0.80	0.80
query15	0.90	0.86	0.86
query16	0.36	0.36	0.35
query17	1.01	0.98	0.99
query18	0.23	0.21	0.21
query19	1.81	1.71	1.76
query20	0.01	0.01	0.01
query21	15.42	0.75	0.65
query22	4.28	6.74	1.97
query23	18.07	1.33	1.38
query24	2.23	0.21	0.22
query25	0.18	0.08	0.09
query26	0.31	0.22	0.21
query27	0.46	0.23	0.23
query28	13.20	1.01	0.98
query29	12.60	3.27	3.29
query30	0.25	0.06	0.05
query31	2.86	0.40	0.40
query32	3.24	0.49	0.48
query33	2.92	2.94	2.89
query34	15.44	4.26	4.26
query35	4.30	4.31	4.29
query36	0.67	0.47	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.03
query40	0.16	0.12	0.14
query41	0.09	0.04	0.05
query42	0.05	0.06	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.07 s
Total hot run time: 30.64 s

@feiniaofeiafei
Copy link
Contributor Author

run cloud_p0

starocean999
starocean999 previously approved these changes Aug 6, 2024
@feiniaofeiafei feiniaofeiafei marked this pull request as draft August 6, 2024 04:15
@feiniaofeiafei feiniaofeiafei marked this pull request as ready for review August 6, 2024 05:53
@feiniaofeiafei feiniaofeiafei marked this pull request as draft August 6, 2024 06:08
@feiniaofeiafei feiniaofeiafei dismissed stale reviews from starocean999 and 924060929 via e5118a6 August 6, 2024 06:55
@feiniaofeiafei
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41542 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e5118a617a05e9764a0d9000855fe24ac61ad6d8, data reload: false

------ Round 1 ----------------------------------
q1	17648	4058	4044	4044
q2	2032	199	211	199
q3	10586	1249	1352	1249
q4	10268	787	881	787
q5	7603	2951	2947	2947
q6	215	138	140	138
q7	1050	617	613	613
q8	9429	1782	1916	1782
q9	8764	6606	6536	6536
q10	8780	3853	3837	3837
q11	436	248	257	248
q12	463	233	234	233
q13	17762	2942	2931	2931
q14	273	238	249	238
q15	539	489	489	489
q16	520	386	385	385
q17	953	936	907	907
q18	8066	7357	7370	7357
q19	1435	1202	1221	1202
q20	554	339	338	338
q21	5264	4798	4822	4798
q22	342	286	284	284
Total cold run time: 112982 ms
Total hot run time: 41542 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4071	4007	4024	4007
q2	330	232	226	226
q3	2981	2992	3155	2992
q4	2005	1978	1981	1978
q5	5587	5477	5449	5449
q6	223	143	142	142
q7	2111	1754	1820	1754
q8	3308	3371	3329	3329
q9	8678	8657	8850	8657
q10	3933	4045	3883	3883
q11	559	478	463	463
q12	785	642	584	584
q13	15098	3093	3131	3093
q14	303	270	271	270
q15	531	481	491	481
q16	468	426	422	422
q17	1763	1720	1751	1720
q18	8230	7811	7789	7789
q19	1766	1723	1749	1723
q20	2047	1857	1851	1851
q21	5717	5533	5306	5306
q22	541	464	471	464
Total cold run time: 71035 ms
Total hot run time: 56583 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169209 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e5118a617a05e9764a0d9000855fe24ac61ad6d8, data reload: false

query1	912	378	370	370
query2	6479	1741	1714	1714
query3	6666	210	219	210
query4	20282	17349	17126	17126
query5	3660	537	504	504
query6	269	175	176	175
query7	4601	304	294	294
query8	259	195	197	195
query9	8504	2401	2393	2393
query10	428	274	287	274
query11	10597	10182	9982	9982
query12	125	87	89	87
query13	1634	381	373	373
query14	9320	7038	6815	6815
query15	216	162	167	162
query16	6942	463	443	443
query17	958	568	559	559
query18	1799	292	289	289
query19	214	152	148	148
query20	93	86	89	86
query21	203	101	106	101
query22	4045	4046	3980	3980
query23	33796	33669	33406	33406
query24	10297	3160	3114	3114
query25	708	449	431	431
query26	1700	161	160	160
query27	2976	291	292	291
query28	7388	2064	2037	2037
query29	1255	520	441	441
query30	227	157	155	155
query31	945	752	805	752
query32	101	52	61	52
query33	669	355	327	327
query34	1001	505	513	505
query35	898	767	771	767
query36	1043	919	871	871
query37	286	86	82	82
query38	2989	2771	2770	2770
query39	878	839	829	829
query40	251	112	110	110
query41	47	43	42	42
query42	127	101	103	101
query43	480	442	424	424
query44	1186	751	728	728
query45	211	175	180	175
query46	1096	792	786	786
query47	1815	1741	1717	1717
query48	358	293	286	286
query49	933	431	440	431
query50	884	432	433	432
query51	6729	6697	6543	6543
query52	104	93	89	89
query53	261	184	178	178
query54	623	447	472	447
query55	76	73	75	73
query56	299	264	252	252
query57	1145	1010	1052	1010
query58	274	274	271	271
query59	2580	2374	2357	2357
query60	304	269	269	269
query61	95	98	100	98
query62	884	666	652	652
query63	208	176	182	176
query64	5662	1918	1885	1885
query65	3173	3106	3123	3106
query66	1299	326	338	326
query67	15338	14943	15026	14943
query68	4346	566	560	560
query69	472	307	293	293
query70	1111	1095	1047	1047
query71	444	276	275	275
query72	7741	2693	2494	2494
query73	780	337	329	329
query74	6055	5691	5646	5646
query75	3661	2717	2706	2706
query76	2806	1175	1273	1175
query77	639	305	309	305
query78	9413	8890	8831	8831
query79	2364	543	533	533
query80	1264	506	510	506
query81	558	228	230	228
query82	786	131	131	131
query83	321	170	170	170
query84	280	81	79	79
query85	2027	327	308	308
query86	459	293	310	293
query87	3269	3150	3104	3104
query88	3850	2435	2414	2414
query89	398	293	295	293
query90	1921	196	195	195
query91	127	100	104	100
query92	57	48	49	48
query93	2558	622	629	622
query94	954	298	299	298
query95	397	276	273	273
query96	619	290	284	284
query97	3233	3051	3045	3045
query98	223	197	197	197
query99	1645	1302	1250	1250
Total cold run time: 266583 ms
Total hot run time: 169209 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e5118a617a05e9764a0d9000855fe24ac61ad6d8, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.05	0.04
query3	0.23	0.04	0.05
query4	1.68	0.06	0.06
query5	0.48	0.49	0.49
query6	1.17	0.73	0.72
query7	0.03	0.02	0.01
query8	0.05	0.05	0.05
query9	0.59	0.52	0.50
query10	0.57	0.59	0.58
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.62	0.62	0.59
query14	0.78	0.79	0.81
query15	0.90	0.87	0.88
query16	0.34	0.34	0.35
query17	0.99	0.96	1.01
query18	0.22	0.21	0.22
query19	1.86	1.74	1.80
query20	0.01	0.01	0.02
query21	15.40	0.77	0.68
query22	4.10	8.22	1.21
query23	18.06	1.35	1.36
query24	2.27	0.22	0.22
query25	0.18	0.08	0.08
query26	0.33	0.22	0.22
query27	0.47	0.24	0.23
query28	13.15	1.00	0.97
query29	12.52	3.30	3.29
query30	0.26	0.05	0.06
query31	2.87	0.41	0.41
query32	3.24	0.51	0.48
query33	2.93	2.94	2.93
query34	15.48	4.25	4.27
query35	4.30	4.29	4.30
query36	0.68	0.47	0.48
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 107.93 s
Total hot run time: 30 s

@feiniaofeiafei feiniaofeiafei marked this pull request as ready for review August 6, 2024 09:47
@feiniaofeiafei
Copy link
Contributor Author

run cloud_p1

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@starocean999 starocean999 merged commit 556c2fb into apache:master Aug 6, 2024
31 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Aug 6, 2024
…8897)

introduced by #38025
Or expression evaluate wrongly in #38025. For example, partition column
is a, b, predicate is a=a or b>1 , partition column range is min<a<5, b
is infinite. The evaluate result of or expression should be b is
infinite(because when a is not null, a=a is true, there is no restrict
for b, e.g. a=1, b=0 satisfy a=a or b>1). #38025 pr has wrongly evaluate
result b >1
feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Aug 7, 2024
…ache#38897)

introduced by apache#38025
Or expression evaluate wrongly in apache#38025. For example, partition column
is a, b, predicate is a=a or b>1 , partition column range is min<a<5, b
is infinite. The evaluate result of or expression should be b is
infinite(because when a is not null, a=a is true, there is no restrict
for b, e.g. a=1, b=0 satisfy a=a or b>1). apache#38025 pr has wrongly evaluate
result b >1
@morrySnow morrySnow removed the doing label Aug 7, 2024
yiguolei pushed a commit that referenced this pull request Aug 7, 2024
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged p0_b reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants