Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](nereids) enable runtime filter use cte as target #40815

Merged
merged 3 commits into from
Sep 19, 2024
Merged

[opt](nereids) enable runtime filter use cte as target #40815

merged 3 commits into from
Sep 19, 2024

Conversation

englefly
Copy link
Contributor

Proposed changes

  1. remove some unused maps in RuntimeFilterContext
  2. let runtime filter use cte as target

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42827 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9a25bc5d46f186432e2493996a1741c34169f400, data reload: false

------ Round 1 ----------------------------------
q1	17574	7273	7141	7141
q2	2014	184	174	174
q3	10753	1261	1406	1261
q4	10364	934	1039	934
q5	7726	3176	3157	3157
q6	246	156	152	152
q7	1037	639	625	625
q8	9463	2047	2036	2036
q9	6740	6319	6329	6319
q10	7024	2544	2536	2536
q11	439	256	256	256
q12	404	235	231	231
q13	17754	3002	3039	3002
q14	288	251	254	251
q15	598	529	526	526
q16	521	435	421	421
q17	976	943	960	943
q18	7348	6855	6817	6817
q19	1376	1243	1225	1225
q20	608	338	329	329
q21	3906	3560	3492	3492
q22	1111	1013	999	999
Total cold run time: 108270 ms
Total hot run time: 42827 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7200	7157	7168	7157
q2	349	244	246	244
q3	3070	3126	3051	3051
q4	2089	2114	2026	2026
q5	5698	5604	5649	5604
q6	245	148	149	148
q7	2148	1790	1772	1772
q8	3374	3439	3401	3401
q9	8787	8954	8772	8772
q10	3509	3542	3559	3542
q11	575	486	476	476
q12	810	632	638	632
q13	10446	3222	3203	3203
q14	317	280	297	280
q15	582	523	550	523
q16	509	476	480	476
q17	1801	1752	1741	1741
q18	8491	8035	8032	8032
q19	1775	1751	1743	1743
q20	2139	1928	1897	1897
q21	5810	5382	5765	5382
q22	1154	1026	1022	1022
Total cold run time: 70878 ms
Total hot run time: 61124 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 199231 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9a25bc5d46f186432e2493996a1741c34169f400, data reload: false

query1	1252	904	919	904
query2	6356	1722	1760	1722
query3	10666	3813	3878	3813
query4	55755	25595	23832	23832
query5	5116	527	533	527
query6	333	191	181	181
query7	5592	322	300	300
query8	273	224	236	224
query9	6079	2588	2588	2588
query10	417	287	273	273
query11	15934	15575	15526	15526
query12	154	104	100	100
query13	1416	400	386	386
query14	10545	7079	6867	6867
query15	212	186	180	180
query16	7021	517	489	489
query17	1170	611	597	597
query18	1912	306	312	306
query19	202	162	156	156
query20	137	115	113	113
query21	214	111	107	107
query22	4718	4545	4502	4502
query23	34937	33987	33623	33623
query24	6112	3181	3125	3125
query25	518	412	422	412
query26	615	160	158	158
query27	1573	290	291	290
query28	2799	2104	2078	2078
query29	713	437	434	434
query30	227	156	173	156
query31	1002	766	803	766
query32	79	54	54	54
query33	443	319	308	308
query34	877	517	473	473
query35	879	767	746	746
query36	1057	905	923	905
query37	144	88	84	84
query38	3980	3885	3993	3885
query39	1464	1420	1457	1420
query40	211	118	116	116
query41	49	50	48	48
query42	127	95	100	95
query43	482	441	453	441
query44	1193	801	773	773
query45	201	174	176	174
query46	1112	860	819	819
query47	1911	1808	1820	1808
query48	379	300	293	293
query49	742	458	450	450
query50	944	446	443	443
query51	7116	7037	6945	6945
query52	99	90	86	86
query53	259	182	182	182
query54	574	474	459	459
query55	75	75	76	75
query56	282	269	281	269
query57	1204	1088	1115	1088
query58	229	242	251	242
query59	2761	2648	2395	2395
query60	293	279	273	273
query61	102	114	102	102
query62	798	665	661	661
query63	225	188	185	185
query64	1601	701	688	688
query65	3283	3184	3175	3175
query66	672	296	300	296
query67	16011	15747	15378	15378
query68	2022	869	856	856
query69	415	335	326	326
query70	1149	1170	1126	1126
query71	360	349	346	346
query72	6063	3423	3366	3366
query73	599	583	585	583
query74	9347	9018	9228	9018
query75	3114	3012	3010	3010
query76	963	858	840	840
query77	529	405	415	405
query78	9463	9355	9358	9355
query79	923	892	882	882
query80	826	824	813	813
query81	463	268	267	267
query82	267	265	267	265
query83	192	237	190	190
query84	225	106	107	106
query85	646	410	397	397
query86	305	319	324	319
query87	4423	4325	4309	4309
query88	4154	4049	4038	4038
query89	378	370	368	368
query90	1713	314	305	305
query91	122	121	165	121
query92	79	75	76	75
query93	1048	1042	1048	1042
query94	638	369	362	362
query95	455	429	425	425
query96	470	472	471	471
query97	3139	3140	3149	3140
query98	234	231	220	220
query99	1670	1323	1316	1316
Total cold run time: 292222 ms
Total hot run time: 199231 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9a25bc5d46f186432e2493996a1741c34169f400, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.07	0.07
query5	0.49	0.50	0.50
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.57	0.52	0.51
query10	0.57	0.59	0.57
query11	0.17	0.12	0.11
query12	0.15	0.13	0.12
query13	0.63	0.61	0.60
query14	1.46	1.47	1.48
query15	0.92	0.88	0.88
query16	0.36	0.37	0.36
query17	1.00	1.02	1.02
query18	0.16	0.16	0.16
query19	1.94	1.85	1.83
query20	0.01	0.01	0.01
query21	15.39	0.69	0.68
query22	4.12	8.80	1.08
query23	18.02	1.28	1.38
query24	2.28	0.23	0.22
query25	0.18	0.09	0.08
query26	0.30	0.19	0.19
query27	0.09	0.08	0.08
query28	13.18	1.14	1.10
query29	12.50	3.36	3.30
query30	0.26	0.05	0.05
query31	2.87	0.43	0.43
query32	3.22	0.49	0.50
query33	3.05	3.01	3.09
query34	15.46	4.34	4.31
query35	4.34	4.34	4.32
query36	0.70	0.49	0.50
query37	0.19	0.17	0.17
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.14	0.14
query41	0.10	0.06	0.05
query42	0.07	0.05	0.05
query43	0.05	0.05	0.04
Total cold run time: 108.4 s
Total hot run time: 30.86 s

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 42956 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit baeaad601e1656541a87091a87f0eca86030b002, data reload: false

------ Round 1 ----------------------------------
q1	17587	7354	7222	7222
q2	2027	208	181	181
q3	10640	1284	1360	1284
q4	10465	988	1000	988
q5	7740	3173	3146	3146
q6	238	153	149	149
q7	1033	625	617	617
q8	9456	2067	2007	2007
q9	6834	6304	6314	6304
q10	7027	2538	2501	2501
q11	434	254	250	250
q12	409	224	223	223
q13	17760	3035	3022	3022
q14	284	256	246	246
q15	590	529	528	528
q16	515	424	439	424
q17	982	950	959	950
q18	7610	6873	6914	6873
q19	1382	1247	1236	1236
q20	618	347	324	324
q21	3861	3528	3470	3470
q22	1112	1011	1047	1011
Total cold run time: 108604 ms
Total hot run time: 42956 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7168	7167	7146	7146
q2	356	241	241	241
q3	3102	3111	3063	3063
q4	2100	2121	2089	2089
q5	5714	5614	5725	5614
q6	246	148	146	146
q7	2190	1772	1811	1772
q8	3365	3416	3394	3394
q9	8818	8859	8777	8777
q10	3457	3583	3596	3583
q11	580	480	477	477
q12	835	633	624	624
q13	7725	3184	3176	3176
q14	305	281	284	281
q15	592	537	549	537
q16	533	460	475	460
q17	1795	1761	1769	1761
q18	8678	8042	8183	8042
q19	1775	1745	1750	1745
q20	2122	1870	1857	1857
q21	5846	5556	5650	5556
q22	1127	996	1057	996
Total cold run time: 68429 ms
Total hot run time: 61337 ms

@englefly
Copy link
Contributor Author

run performance

@englefly
Copy link
Contributor Author

run p0

@englefly
Copy link
Contributor Author

run cloud_p0

@doris-robot
Copy link

TPC-H: Total hot run time: 42990 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit baeaad601e1656541a87091a87f0eca86030b002, data reload: false

------ Round 1 ----------------------------------
q1	17583	7329	7261	7261
q2	2041	187	179	179
q3	10548	1305	1376	1305
q4	10504	977	1071	977
q5	7722	3195	3158	3158
q6	248	159	154	154
q7	1053	642	617	617
q8	9475	2054	2049	2049
q9	6844	6322	6367	6322
q10	7057	2557	2546	2546
q11	438	249	256	249
q12	413	231	230	230
q13	17770	3061	3034	3034
q14	294	246	252	246
q15	599	533	519	519
q16	516	422	429	422
q17	993	936	959	936
q18	7395	6673	6829	6673
q19	1375	1240	1228	1228
q20	621	341	325	325
q21	3944	3552	3547	3547
q22	1111	1013	1046	1013
Total cold run time: 108544 ms
Total hot run time: 42990 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7213	7225	7268	7225
q2	363	239	242	239
q3	3086	3087	3027	3027
q4	2071	2147	2075	2075
q5	5668	5623	5703	5623
q6	245	155	156	155
q7	2196	1786	1773	1773
q8	3411	3475	3420	3420
q9	8862	8955	8822	8822
q10	3515	3558	3606	3558
q11	578	480	474	474
q12	845	615	645	615
q13	9185	3190	3233	3190
q14	333	273	284	273
q15	576	526	554	526
q16	510	475	473	473
q17	1804	1769	1767	1767
q18	8530	8177	8058	8058
q19	1776	1766	1752	1752
q20	2140	1911	1885	1885
q21	5988	5461	5710	5461
q22	1155	1079	1036	1036
Total cold run time: 70050 ms
Total hot run time: 61427 ms

@englefly
Copy link
Contributor Author

run fe-ut

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 43268 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit baeaad601e1656541a87091a87f0eca86030b002, data reload: false

------ Round 1 ----------------------------------
q1	18211	7564	7353	7353
q2	2418	219	211	211
q3	10731	1442	1444	1442
q4	11025	1056	998	998
q5	8463	3324	3222	3222
q6	247	159	153	153
q7	1060	642	624	624
q8	9516	2064	2053	2053
q9	6810	6334	6316	6316
q10	7016	2509	2510	2509
q11	431	246	257	246
q12	416	227	223	223
q13	17764	3028	3024	3024
q14	289	247	248	247
q15	599	532	530	530
q16	519	425	422	422
q17	995	948	953	948
q18	7531	6668	6864	6668
q19	1383	1243	1250	1243
q20	619	339	336	336
q21	3890	3531	3525	3525
q22	1066	975	1019	975
Total cold run time: 110999 ms
Total hot run time: 43268 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7178	7198	7135	7135
q2	343	239	235	235
q3	2935	2887	2930	2887
q4	1946	1943	1982	1943
q5	5445	5422	5496	5422
q6	239	142	142	142
q7	2026	1666	1667	1666
q8	3292	3337	3380	3337
q9	8507	8445	8463	8445
q10	3390	3456	3461	3456
q11	579	461	463	461
q12	744	584	582	582
q13	5804	3039	3012	3012
q14	300	287	283	283
q15	576	530	519	519
q16	483	450	441	441
q17	1767	1755	1733	1733
q18	8097	7676	7718	7676
q19	1734	1710	1702	1702
q20	2057	1784	1791	1784
q21	5805	5415	5389	5389
q22	1116	972	988	972
Total cold run time: 64363 ms
Total hot run time: 59222 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195795 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit baeaad601e1656541a87091a87f0eca86030b002, data reload: false

query1	937	399	378	378
query2	6468	1774	1774	1774
query3	6661	219	222	219
query4	26669	24092	24101	24092
query5	4994	546	527	527
query6	262	194	167	167
query7	4579	305	308	305
query8	279	225	229	225
query9	8497	2604	2599	2599
query10	450	286	272	272
query11	16978	15489	15574	15489
query12	176	103	104	103
query13	1710	404	398	398
query14	11205	6907	7164	6907
query15	214	176	182	176
query16	7589	487	465	465
query17	1544	591	586	586
query18	2000	310	301	301
query19	204	159	163	159
query20	119	112	110	110
query21	218	105	104	104
query22	4536	4342	4025	4025
query23	34583	33833	33835	33833
query24	9696	3189	3057	3057
query25	690	418	423	418
query26	1385	159	160	159
query27	2852	287	295	287
query28	6789	2109	2085	2085
query29	966	434	436	434
query30	293	166	155	155
query31	1011	810	813	810
query32	111	61	62	61
query33	724	312	302	302
query34	916	481	496	481
query35	886	764	756	756
query36	1056	917	898	898
query37	175	88	86	86
query38	4034	3990	3991	3990
query39	1620	1584	1406	1406
query40	286	119	118	118
query41	51	49	50	49
query42	119	99	104	99
query43	499	459	428	428
query44	1259	816	785	785
query45	200	172	173	172
query46	1120	845	822	822
query47	1884	1775	1823	1775
query48	369	305	300	300
query49	1154	469	462	462
query50	916	440	450	440
query51	7122	6935	6940	6935
query52	105	88	90	88
query53	261	188	188	188
query54	771	476	481	476
query55	81	77	80	77
query56	286	291	296	291
query57	1223	1102	1110	1102
query58	271	237	247	237
query59	2754	2537	2488	2488
query60	304	281	277	277
query61	127	226	103	103
query62	925	692	698	692
query63	230	187	181	181
query64	5287	706	660	660
query65	3279	3185	3194	3185
query66	1389	297	291	291
query67	15951	15606	15388	15388
query68	3195	876	875	875
query69	438	336	325	325
query70	1186	1154	1114	1114
query71	367	349	345	345
query72	5959	3390	3380	3380
query73	603	591	586	586
query74	9175	9322	9265	9265
query75	3155	3034	3025	3025
query76	1961	869	873	869
query77	432	400	409	400
query78	9431	9253	9367	9253
query79	935	902	877	877
query80	879	853	827	827
query81	444	271	269	269
query82	267	263	263	263
query83	193	192	193	192
query84	237	107	107	107
query85	658	459	402	402
query86	309	311	336	311
query87	4534	4379	4370	4370
query88	4220	4187	4163	4163
query89	392	373	376	373
query90	1325	325	322	322
query91	124	126	158	126
query92	76	74	77	74
query93	1060	1083	1068	1068
query94	605	378	387	378
query95	452	424	421	421
query96	492	485	486	485
query97	3152	3114	3118	3114
query98	224	231	222	222
query99	1561	1321	1286	1286
Total cold run time: 280340 ms
Total hot run time: 195795 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit baeaad601e1656541a87091a87f0eca86030b002, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.04
query4	1.68	0.07	0.06
query5	0.50	0.50	0.51
query6	1.15	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.51
query10	0.58	0.57	0.56
query11	0.17	0.12	0.12
query12	0.17	0.12	0.12
query13	0.62	0.60	0.60
query14	1.45	1.48	1.45
query15	0.89	0.87	0.88
query16	0.36	0.36	0.35
query17	1.03	1.02	1.01
query18	0.22	0.20	0.20
query19	1.91	1.82	1.79
query20	0.01	0.01	0.00
query21	15.39	0.66	0.66
query22	4.05	7.61	1.15
query23	17.89	1.34	1.38
query24	2.23	0.22	0.22
query25	0.18	0.08	0.08
query26	0.29	0.18	0.19
query27	0.08	0.07	0.08
query28	13.18	1.16	1.11
query29	12.53	3.36	3.35
query30	0.24	0.06	0.05
query31	2.88	0.42	0.42
query32	3.22	0.49	0.50
query33	3.04	3.07	3.06
query34	15.44	4.34	4.31
query35	4.36	4.40	4.34
query36	0.69	0.48	0.50
query37	0.18	0.16	0.16
query38	0.17	0.14	0.14
query39	0.05	0.04	0.04
query40	0.18	0.13	0.13
query41	0.10	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 108.2 s
Total hot run time: 30.95 s

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41796 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 39c8c1b3b6266fe4c93b48e1f64d7de0bdf8f725, data reload: false

------ Round 1 ----------------------------------
q1	18047	7671	7309	7309
q2	2534	161	160	160
q3	11502	1160	1283	1160
q4	11393	742	724	724
q5	8109	3147	3128	3128
q6	246	152	150	150
q7	997	616	610	610
q8	9451	2054	2021	2021
q9	6845	6422	6428	6422
q10	7001	2270	2323	2270
q11	434	263	255	255
q12	408	221	211	211
q13	17794	2942	2992	2942
q14	233	225	227	225
q15	578	539	518	518
q16	677	612	623	612
q17	973	794	760	760
q18	7381	6797	6802	6797
q19	1393	1098	1016	1016
q20	579	280	294	280
q21	3975	3292	3239	3239
q22	1105	987	1022	987
Total cold run time: 111655 ms
Total hot run time: 41796 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7247	7249	7195	7195
q2	329	240	226	226
q3	2891	2732	2806	2732
q4	1953	1721	1712	1712
q5	5366	5379	5406	5379
q6	232	137	137	137
q7	2062	1684	1715	1684
q8	3172	3307	3307	3307
q9	8408	8428	8381	8381
q10	3360	3369	3348	3348
q11	561	473	470	470
q12	778	595	600	595
q13	6498	2997	2963	2963
q14	282	263	258	258
q15	561	515	511	511
q16	702	653	663	653
q17	1752	1535	1517	1517
q18	7828	7247	7339	7247
q19	1657	1558	1634	1558
q20	2030	1819	1825	1819
q21	5312	5163	5156	5156
q22	1109	1038	1000	1000
Total cold run time: 64090 ms
Total hot run time: 57848 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 194558 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 39c8c1b3b6266fe4c93b48e1f64d7de0bdf8f725, data reload: false

query1	958	366	367	366
query2	6516	2009	1984	1984
query3	6713	219	224	219
query4	34122	23410	23465	23410
query5	4313	493	461	461
query6	251	163	156	156
query7	4618	311	301	301
query8	271	217	207	207
query9	9668	2666	2664	2664
query10	444	283	275	275
query11	18112	15081	15254	15081
query12	150	100	99	99
query13	1651	416	408	408
query14	10243	7110	7583	7110
query15	267	170	174	170
query16	8021	461	468	461
query17	1675	580	560	560
query18	2114	311	308	308
query19	370	145	146	145
query20	118	109	108	108
query21	214	104	105	104
query22	4608	4482	4374	4374
query23	34688	34005	33922	33922
query24	11284	2928	2893	2893
query25	650	416	412	412
query26	1193	158	159	158
query27	2740	285	288	285
query28	8187	2478	2450	2450
query29	845	430	427	427
query30	318	163	159	159
query31	1014	799	808	799
query32	97	55	58	55
query33	775	322	307	307
query34	932	477	501	477
query35	879	716	731	716
query36	1065	918	904	904
query37	159	85	82	82
query38	4070	3890	3901	3890
query39	1479	1422	1430	1422
query40	207	101	100	100
query41	51	47	52	47
query42	115	97	98	97
query43	519	475	470	470
query44	1321	827	803	803
query45	197	163	167	163
query46	1142	740	752	740
query47	1943	1836	1815	1815
query48	455	356	365	356
query49	1134	422	398	398
query50	831	401	412	401
query51	7050	6903	6928	6903
query52	100	91	90	90
query53	254	182	184	182
query54	1377	465	470	465
query55	80	80	80	80
query56	297	266	263	263
query57	1237	1104	1115	1104
query58	250	252	270	252
query59	3133	2927	3055	2927
query60	315	269	277	269
query61	128	126	126	126
query62	838	751	677	677
query63	225	186	185	185
query64	5276	650	657	650
query65	3286	3181	3160	3160
query66	1442	304	324	304
query67	15948	15521	15616	15521
query68	3226	843	831	831
query69	443	345	330	330
query70	1192	1191	1149	1149
query71	345	324	324	324
query72	6106	3362	3434	3362
query73	584	575	566	566
query74	9656	9054	8928	8928
query75	3072	2881	2895	2881
query76	1911	849	841	841
query77	403	350	353	350
query78	9470	9751	9281	9281
query79	909	884	875	875
query80	577	554	559	554
query81	444	253	247	247
query82	231	229	226	226
query83	163	158	160	158
query84	239	109	106	106
query85	662	363	352	352
query86	308	298	310	298
query87	4439	4400	4414	4400
query88	4843	4038	3994	3994
query89	373	368	353	353
query90	1316	314	308	308
query91	168	162	163	162
query92	76	71	72	71
query93	890	887	875	875
query94	508	383	344	344
query95	426	396	419	396
query96	474	485	480	480
query97	3127	3119	3133	3119
query98	224	235	234	234
query99	1410	1313	1286	1286
Total cold run time: 293158 ms
Total hot run time: 194558 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 32.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 39c8c1b3b6266fe4c93b48e1f64d7de0bdf8f725, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.02
query3	0.23	0.06	0.05
query4	1.65	0.10	0.10
query5	0.50	0.52	0.53
query6	1.16	0.73	0.74
query7	0.02	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.49
query10	0.55	0.55	0.56
query11	0.14	0.11	0.11
query12	0.13	0.10	0.11
query13	0.60	0.59	0.60
query14	2.97	3.03	2.96
query15	0.88	0.83	0.82
query16	0.38	0.39	0.37
query17	1.04	1.05	1.00
query18	0.22	0.20	0.20
query19	1.91	1.92	1.97
query20	0.01	0.01	0.01
query21	15.35	0.59	0.59
query22	2.41	2.22	1.85
query23	17.29	0.84	0.84
query24	2.50	1.27	0.80
query25	0.24	0.09	0.04
query26	0.44	0.14	0.14
query27	0.04	0.04	0.03
query28	11.13	1.09	1.07
query29	12.57	3.26	3.25
query30	0.25	0.07	0.06
query31	2.87	0.38	0.37
query32	3.29	0.47	0.46
query33	2.97	2.99	3.00
query34	17.09	4.46	4.43
query35	4.40	4.41	4.45
query36	0.68	0.49	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.02	0.02
query40	0.15	0.12	0.12
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 107.08 s
Total hot run time: 32.72 s

morrySnow
morrySnow previously approved these changes Sep 18, 2024
View reviewed changes
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Sep 18, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Sep 18, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41571 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c9fde7b4e380d877ff49f56910f0b9e47c3c34c, data reload: false

------ Round 1 ----------------------------------
q1	17590	8051	7275	7275
q2	2056	165	159	159
q3	10624	1165	1250	1165
q4	10219	770	696	696
q5	7747	3144	3103	3103
q6	231	145	150	145
q7	1008	620	605	605
q8	9428	2101	2036	2036
q9	8217	6427	6431	6427
q10	7092	2297	2273	2273
q11	451	249	250	249
q12	499	217	217	217
q13	17776	3006	3029	3006
q14	249	218	211	211
q15	567	507	522	507
q16	697	607	620	607
q17	1011	871	819	819
q18	7454	6834	6701	6701
q19	1391	1010	925	925
q20	574	294	279	279
q21	4083	3282	3168	3168
q22	1094	1025	998	998
Total cold run time: 110058 ms
Total hot run time: 41571 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7225	7221	7227	7221
q2	328	225	226	225
q3	2961	2774	2841	2774
q4	1975	1685	1724	1685
q5	5428	5397	5437	5397
q6	225	137	136	136
q7	2125	1712	1710	1710
q8	3175	3326	3340	3326
q9	8414	8469	8405	8405
q10	3394	3371	3382	3371
q11	575	477	482	477
q12	767	617	590	590
q13	11705	3010	2987	2987
q14	291	267	248	248
q15	566	510	517	510
q16	706	669	670	669
q17	1777	1541	1530	1530
q18	7664	7363	7227	7227
q19	1665	1511	1575	1511
q20	2044	1816	1818	1816
q21	5358	5313	5087	5087
q22	1104	1039	993	993
Total cold run time: 69472 ms
Total hot run time: 57895 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 195832 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c9fde7b4e380d877ff49f56910f0b9e47c3c34c, data reload: false

query1	949	382	375	375
query2	6372	2049	2063	2049
query3	6305	215	218	215
query4	34270	23631	23596	23596
query5	3467	482	491	482
query6	265	180	179	179
query7	4180	300	304	300
query8	294	227	221	221
query9	9208	2687	2663	2663
query10	467	297	298	297
query11	18407	15341	15442	15341
query12	152	98	98	98
query13	1365	423	394	394
query14	10123	7962	7623	7623
query15	307	173	175	173
query16	7492	482	465	465
query17	1612	599	567	567
query18	1984	319	315	315
query19	330	159	159	159
query20	117	107	109	107
query21	184	112	106	106
query22	4437	4372	4212	4212
query23	34758	34012	34552	34012
query24	9893	3029	2863	2863
query25	636	408	422	408
query26	1135	165	166	165
query27	2204	298	297	297
query28	7232	2470	2457	2457
query29	775	436	432	432
query30	322	176	154	154
query31	1016	822	809	809
query32	88	60	61	60
query33	709	322	306	306
query34	940	492	499	492
query35	870	741	734	734
query36	1079	929	957	929
query37	166	88	88	88
query38	4112	3878	3884	3878
query39	1486	1441	1431	1431
query40	202	102	99	99
query41	53	49	49	49
query42	121	97	102	97
query43	531	469	490	469
query44	1227	824	807	807
query45	197	167	172	167
query46	1155	804	775	775
query47	1869	1778	1827	1778
query48	445	360	372	360
query49	1101	426	407	407
query50	823	408	430	408
query51	7068	7062	6860	6860
query52	107	88	88	88
query53	260	188	191	188
query54	766	469	466	466
query55	78	78	77	77
query56	282	273	266	266
query57	1191	1080	1097	1080
query58	240	242	263	242
query59	3131	2886	2857	2857
query60	298	280	286	280
query61	110	100	105	100
query62	833	659	680	659
query63	227	191	186	186
query64	3798	666	654	654
query65	3290	3216	3176	3176
query66	726	295	301	295
query67	15930	15629	15480	15480
query68	2501	878	865	865
query69	446	343	346	343
query70	1177	1228	1190	1190
query71	367	325	337	325
query72	5822	3416	3390	3390
query73	585	584	589	584
query74	9466	8971	9155	8971
query75	3012	2889	2928	2889
query76	1800	898	896	896
query77	450	359	379	359
query78	9402	9227	9430	9227
query79	930	891	882	882
query80	602	589	578	578
query81	466	259	254	254
query82	235	229	229	229
query83	157	168	163	163
query84	245	110	99	99
query85	679	379	355	355
query86	324	330	315	315
query87	4389	4390	4322	4322
query88	4469	4079	4039	4039
query89	373	376	372	372
query90	1437	313	306	306
query91	164	174	169	169
query92	78	77	74	74
query93	910	905	895	895
query94	551	389	390	389
query95	432	411	413	411
query96	489	482	484	482
query97	3084	3110	3192	3110
query98	223	223	253	223
query99	1440	1304	1305	1304
Total cold run time: 282472 ms
Total hot run time: 195832 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 33.04 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c9fde7b4e380d877ff49f56910f0b9e47c3c34c, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.64	0.10	0.10
query5	0.54	0.50	0.52
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.57	0.50	0.50
query10	0.54	0.55	0.54
query11	0.15	0.11	0.10
query12	0.15	0.11	0.11
query13	0.60	0.59	0.58
query14	3.06	2.99	3.00
query15	0.88	0.83	0.82
query16	0.39	0.39	0.37
query17	0.99	1.05	1.01
query18	0.19	0.19	0.20
query19	1.97	1.90	1.89
query20	0.01	0.01	0.01
query21	15.36	0.60	0.61
query22	2.27	2.86	2.49
query23	17.10	0.90	0.81
query24	2.66	0.55	1.51
query25	0.24	0.10	0.11
query26	0.39	0.15	0.14
query27	0.05	0.04	0.04
query28	10.93	1.10	1.06
query29	12.56	3.30	3.23
query30	0.25	0.06	0.06
query31	2.87	0.38	0.37
query32	3.27	0.46	0.46
query33	3.01	3.00	2.99
query34	17.03	4.39	4.35
query35	4.48	4.42	4.41
query36	0.66	0.49	0.48
query37	0.08	0.06	0.05
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.13	0.12
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.04	0.04
Total cold run time: 106.81 s
Total hot run time: 33.04 s

@englefly
Copy link
Contributor Author

run p0

1 similar comment
@englefly
Copy link
Contributor Author

run p0

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Sep 19, 2024
@englefly englefly merged commit 7caa3b9 into apache:master Sep 19, 2024
26 of 29 checks passed
@englefly englefly deleted the rf-cte branch September 19, 2024 03:29
@englefly englefly mentioned this pull request Sep 22, 2024
Merged
englefly added a commit that referenced this pull request Sep 23, 2024
@englefly
[opt](nereids) enable runtime filter use cte as target #40815 (2.1) (#…
5bcea19 
…41090)

## Proposed changes
pick #40815
Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Oct 9, 2024
@englefly @dataroaring
[opt](nereids) enable runtime filter use cte as target (#40815)
0fec418 
## Proposed changes
1. remove some unused maps in RuntimeFilterContext
2. let runtime filter use cte as target

Issue Number: close #xxx

<!--Describe your changes.-->
@yiguolei yiguolei mentioned this pull request Nov 6, 2024
Open
@morrySnow morrySnow mentioned this pull request Jan 2, 2025
Merged
16 tasks
@englefly englefly mentioned this pull request Jan 3, 2025
Merged
16 tasks
englefly added a commit that referenced this pull request Jan 4, 2025
@englefly
[fix](nereids) Create Runtime filter even if equal.right is not in al…
f4c3e39 
…iasTransferMap-branch3.0 (#46224)

### What problem does this PR solve?

Issue Number: close #xxx
pick #46406
Related PR: #40815
englefly added a commit that referenced this pull request Jan 4, 2025
@englefly
[fix](nereids) Create Runtime filter even if equal.right is not in al…
2358dc4 
…iasTransferMap (branch-2.1) (#46407)

### What problem does this PR solve?

pick #46406
Related PR: #40815
englefly added a commit that referenced this pull request Jan 6, 2025
@englefly
[fix](nereids) Create Runtime filter even if equal.right is not in al…
a539e22 
…iasTransferMap (#46406)

### What problem does this PR solve?
fix a bug introduced by #40815
@yiguolei yiguolei mentioned this pull request Jan 19, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@starocean999 starocean999 starocean999 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.7-merged dev/3.0.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@englefly @doris-robot @starocean999 @morrySnow