[feature](nereids) add rewrite rule PushCountIntoUnionAll #33530

feiniaofeiafei · 2024-04-11T07:23:12Z

add transform:

/**
 * LogicalAggregate  (groupByExpr=[c1#13], outputExpr=[c1#13, count(c1#13) AS `count(c1)`#15])
 *  +--LogicalUnion (outputs=[c1#13], regularChildrenOutputs=[[c1#9], [a#4], [a#7]])
 *    |--child1 (output = [[c1#9]])
 *    |--child2 (output = [[a#4]])
 *    +--child3 (output = [[a#7]])
 * transform to:
 * LogicalAggregate (groupByExpr=[c1#13], outputExpr=[c1#13, sum0(count(c1)#19) AS `count(c1)`#15])
 *  +--LogicalUnion (outputs=[c1#13, count(c1)#19], regularChildrenOutputs=[[c1#9, count(c1)#16],
 *   [a#4, count(a)#17], [a#7, count(a)#18]])
 *    |--LogicalAggregate (groupByExpr=[c1#9], outputExpr=[c1#9, count(c1#9) AS `count(c1)`#16])
 *    |  +--child1
 *    |--LogicalAggregate (groupByExpr=[a#4], outputExpr=[a#4, count(a#4) AS `count(a)`#17])
 *    |  +--child2
 *    +--LogicalAggregate (groupByExpr=[a#7], outputExpr=[a#7, count(a#7) AS `count(a)`#18]]
 *      +--child3
 */

doris-robot · 2024-04-11T07:23:17Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

feiniaofeiafei · 2024-04-11T07:39:18Z

run buildall

feiniaofeiafei · 2024-04-11T12:59:16Z

run p0

feiniaofeiafei · 2024-04-15T02:40:42Z

run buildall

feiniaofeiafei · 2024-04-15T08:59:22Z

run buildall

doris-robot · 2024-04-15T09:35:44Z

TPC-H: Total hot run time: 38758 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b24cf6000f01af28cd7c640f923f11635dc47fc, data reload: false

------ Round 1 ----------------------------------
q1	17617	4283	4260	4260
q2	2001	191	188	188
q3	10461	1145	1213	1145
q4	10201	830	833	830
q5	7562	2674	2619	2619
q6	220	134	134	134
q7	1013	621	584	584
q8	9210	2064	2047	2047
q9	7947	6597	6574	6574
q10	8664	3543	3528	3528
q11	467	237	234	234
q12	409	222	227	222
q13	18893	2967	2979	2967
q14	277	234	229	229
q15	518	483	497	483
q16	525	394	384	384
q17	973	708	711	708
q18	7432	6810	6738	6738
q19	6082	1587	1552	1552
q20	709	318	315	315
q21	3715	2704	2742	2704
q22	369	313	330	313
Total cold run time: 115265 ms
Total hot run time: 38758 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4491	4445	4424	4424
q2	383	278	269	269
q3	3133	2933	2882	2882
q4	2007	1695	1729	1695
q5	5478	5419	5369	5369
q6	213	126	126	126
q7	2261	1882	1892	1882
q8	3224	3400	3332	3332
q9	8609	8558	8855	8558
q10	4054	3987	3850	3850
q11	595	515	502	502
q12	799	649	629	629
q13	17079	3207	3146	3146
q14	350	303	286	286
q15	503	486	480	480
q16	521	469	463	463
q17	1826	1510	1497	1497
q18	8288	8027	7614	7614
q19	1720	1617	1644	1617
q20	2092	1878	1854	1854
q21	7535	4922	4933	4922
q22	545	480	477	477
Total cold run time: 75706 ms
Total hot run time: 55874 ms

doris-robot · 2024-04-15T09:47:27Z

TPC-DS: Total hot run time: 183042 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b24cf6000f01af28cd7c640f923f11635dc47fc, data reload: false

query1	1241	1131	1109	1109
query2	6242	2580	2424	2424
query3	6656	206	202	202
query4	37198	21711	21315	21315
query5	4147	394	396	394
query6	237	186	186	186
query7	4040	290	292	290
query8	215	164	164	164
query9	5761	2310	2264	2264
query10	369	233	244	233
query11	14731	14256	14492	14256
query12	135	97	86	86
query13	985	348	355	348
query14	9436	6880	6849	6849
query15	216	181	188	181
query16	7236	263	271	263
query17	1748	585	596	585
query18	1711	287	275	275
query19	202	156	159	156
query20	98	91	90	90
query21	199	130	124	124
query22	5038	4780	4803	4780
query23	34345	33136	33088	33088
query24	11509	3063	3000	3000
query25	544	417	405	405
query26	741	166	160	160
query27	2920	369	367	367
query28	6427	2138	2018	2018
query29	856	650	619	619
query30	253	166	180	166
query31	959	748	751	748
query32	59	55	53	53
query33	555	260	252	252
query34	917	501	521	501
query35	840	707	706	706
query36	1043	905	883	883
query37	111	69	75	69
query38	3511	3290	3304	3290
query39	1104	1019	1015	1015
query40	172	130	129	129
query41	45	42	42	42
query42	102	96	105	96
query43	573	559	537	537
query44	1385	734	744	734
query45	286	273	252	252
query46	1093	766	730	730
query47	2047	1904	1956	1904
query48	376	311	298	298
query49	789	377	372	372
query50	789	398	391	391
query51	6801	6639	6777	6639
query52	100	84	90	84
query53	344	277	279	277
query54	246	225	220	220
query55	79	69	72	69
query56	239	222	227	222
query57	1207	1143	1126	1126
query58	244	215	209	209
query59	3218	3308	3039	3039
query60	247	241	241	241
query61	92	88	85	85
query62	608	456	435	435
query63	304	286	284	284
query64	4001	3929	3607	3607
query65	3099	3050	3005	3005
query66	718	319	316	316
query67	15566	14892	14880	14880
query68	7625	555	562	555
query69	607	331	316	316
query70	1281	1184	1136	1136
query71	466	272	267	267
query72	6437	2634	2437	2437
query73	834	323	318	318
query74	6701	6311	6370	6311
query75	3614	2698	2601	2601
query76	4179	1115	1090	1090
query77	615	240	260	240
query78	10965	10288	10175	10175
query79	7076	522	523	522
query80	1497	430	441	430
query81	538	239	232	232
query82	841	105	138	105
query83	200	166	167	166
query84	262	85	85	85
query85	922	265	263	263
query86	516	273	313	273
query87	3421	3285	3281	3281
query88	5892	2265	2274	2265
query89	487	373	361	361
query90	1819	176	177	176
query91	123	94	93	93
query92	54	45	47	45
query93	6084	515	489	489
query94	986	179	178	178
query95	380	293	281	281
query96	592	272	260	260
query97	3121	2942	2893	2893
query98	239	213	209	209
query99	1177	852	868	852
Total cold run time: 298856 ms
Total hot run time: 183042 ms

doris-robot · 2024-04-15T09:52:59Z

ClickBench: Total hot run time: 30.4 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b24cf6000f01af28cd7c640f923f11635dc47fc, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.07
query5	0.50	0.50	0.50
query6	1.48	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.61	0.58
query14	0.76	0.77	0.80
query15	0.83	0.82	0.81
query16	0.36	0.37	0.37
query17	0.94	0.95	0.98
query18	0.23	0.23	0.23
query19	1.78	1.72	1.80
query20	0.01	0.01	0.01
query21	15.40	0.66	0.64
query22	4.83	7.29	1.96
query23	18.29	1.37	1.20
query24	2.05	0.27	0.22
query25	0.15	0.08	0.08
query26	0.27	0.16	0.16
query27	0.07	0.07	0.08
query28	13.34	1.00	0.99
query29	12.59	3.33	3.30
query30	0.26	0.07	0.06
query31	2.87	0.38	0.37
query32	3.28	0.46	0.47
query33	2.81	2.83	2.78
query34	17.18	4.39	4.49
query35	4.50	4.50	4.49
query36	0.63	0.47	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.18	0.15	0.14
query41	0.11	0.06	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.26 s
Total hot run time: 30.4 s

doris-robot · 2024-04-15T09:56:04Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 1b24cf6000f01af28cd7c640f923f11635dc47fc with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       14.3 seconds inserted 10000000 Rows, about 699K ops/s

924060929 · 2024-04-19T02:50:36Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckAfterRewrite.java

@@ -106,7 +106,7 @@ private void checkAllSlotReferenceFromChildren(Plan plan) {
        if (notFromChildren.isEmpty()) {
            return;
        }
-
+        System.out.println(plan.treeString());


remove this line

feiniaofeiafei · 2024-04-19T02:57:35Z

run buildall

924060929 · 2024-04-19T02:52:25Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushCountIntoUnionAll.java

+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+// copied from https://github.com/trinodb/trino/blob/master/core/trino-main/src/main/java/io/trino/server/PluginManager.java


remove this line

924060929 · 2024-04-19T03:02:04Z

fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/PushCountIntoUnionAll.java

+                newLogicalUnionOutputs.add(new SlotReference(ce.getName(), ce.getDataType(), ce.nullable()));
+            } else if (ce instanceof SlotReference) {
+                newLogicalUnionOutputs.add(ce);
+            }


add else condition and return origin LogicalUnion, means you can not process this cases

feiniaofeiafei · 2024-04-23T02:37:49Z

run buildall

feiniaofeiafei · 2024-04-23T09:46:02Z

run p0

feiniaofeiafei · 2024-04-24T02:58:17Z

run buildall

feiniaofeiafei · 2024-06-05T12:43:18Z

run buildall

doris-robot · 2024-06-05T13:08:21Z

TPC-H: Total hot run time: 39753 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7583d95015619320f6746c19b06710fca42d1238, data reload: false

------ Round 1 ----------------------------------
q1	17852	4546	4397	4397
q2	2643	200	189	189
q3	11677	1076	1192	1076
q4	10298	768	833	768
q5	7540	2738	2677	2677
q6	223	138	139	138
q7	974	603	623	603
q8	9449	2045	2078	2045
q9	8777	6499	6468	6468
q10	8977	3672	3741	3672
q11	446	245	233	233
q12	409	235	223	223
q13	17765	2981	2935	2935
q14	255	214	220	214
q15	508	478	477	477
q16	489	376	376	376
q17	947	591	663	591
q18	8087	7445	7556	7445
q19	3128	1525	1388	1388
q20	671	314	300	300
q21	4915	3200	3819	3200
q22	394	341	338	338
Total cold run time: 116424 ms
Total hot run time: 39753 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4220	4279	4220
q2	366	263	274	263
q3	2974	2788	2756	2756
q4	1902	1581	1637	1581
q5	5232	5252	5271	5252
q6	219	126	126	126
q7	2116	1763	1700	1700
q8	3197	3322	3291	3291
q9	8316	8305	8270	8270
q10	3870	3690	3688	3688
q11	578	492	482	482
q12	774	591	618	591
q13	17573	2990	2989	2989
q14	301	261	275	261
q15	516	477	498	477
q16	478	401	405	401
q17	1772	1478	1465	1465
q18	7605	7417	7473	7417
q19	1672	1460	1638	1460
q20	1981	1773	1822	1773
q21	4913	4684	4599	4599
q22	622	530	516	516
Total cold run time: 71347 ms
Total hot run time: 53578 ms

doris-robot · 2024-06-05T13:19:18Z

TPC-DS: Total hot run time: 171849 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7583d95015619320f6746c19b06710fca42d1238, data reload: false

query1	942	385	373	373
query2	6471	2431	2232	2232
query3	6651	208	211	208
query4	19732	17402	17319	17319
query5	4179	460	446	446
query6	264	168	156	156
query7	4581	302	302	302
query8	310	283	271	271
query9	8461	2390	2358	2358
query10	453	318	276	276
query11	10517	10213	10137	10137
query12	138	82	84	82
query13	1653	362	362	362
query14	9221	6024	7590	6024
query15	238	181	184	181
query16	7793	262	264	262
query17	1849	530	537	530
query18	1978	289	281	281
query19	204	158	162	158
query20	95	83	86	83
query21	210	161	130	130
query22	4454	4145	4074	4074
query23	33726	33167	33116	33116
query24	11600	2855	2757	2757
query25	665	354	356	354
query26	1715	157	152	152
query27	3093	329	327	327
query28	7682	2038	2041	2038
query29	1031	611	598	598
query30	291	149	157	149
query31	953	744	735	735
query32	92	52	54	52
query33	755	278	287	278
query34	971	459	488	459
query35	762	594	615	594
query36	1074	879	937	879
query37	153	68	69	68
query38	2881	2704	2784	2704
query39	893	798	810	798
query40	277	128	122	122
query41	54	51	54	51
query42	115	100	97	97
query43	589	542	536	536
query44	1193	741	745	741
query45	194	164	171	164
query46	1072	695	710	695
query47	1853	1761	1804	1761
query48	380	305	294	294
query49	1140	404	398	398
query50	773	401	385	385
query51	6762	6692	6663	6663
query52	103	92	90	90
query53	361	299	292	292
query54	965	488	450	450
query55	78	74	74	74
query56	275	259	258	258
query57	1146	1062	1030	1030
query58	280	242	238	238
query59	3384	3120	3122	3120
query60	288	278	279	278
query61	93	87	88	87
query62	643	454	425	425
query63	314	288	290	288
query64	9913	2249	1735	1735
query65	3157	3088	3112	3088
query66	1363	332	332	332
query67	15458	14888	14960	14888
query68	4552	548	557	548
query69	444	305	311	305
query70	1124	1119	1111	1111
query71	372	286	290	286
query72	7026	5660	5666	5660
query73	783	323	327	323
query74	6048	5554	5560	5554
query75	3364	2614	2647	2614
query76	2317	897	901	897
query77	433	308	303	303
query78	10213	9951	9630	9630
query79	2426	526	518	518
query80	920	471	464	464
query81	582	224	220	220
query82	700	104	101	101
query83	236	171	167	167
query84	242	85	84	84
query85	1983	277	267	267
query86	509	338	315	315
query87	3332	3057	3086	3057
query88	4352	2465	2457	2457
query89	478	391	394	391
query90	1808	196	193	193
query91	136	109	109	109
query92	68	52	54	52
query93	2336	526	511	511
query94	1306	197	200	197
query95	482	314	317	314
query96	598	273	267	267
query97	3148	3021	3005	3005
query98	210	197	192	192
query99	1155	862	854	854
Total cold run time: 274637 ms
Total hot run time: 171849 ms

doris-robot · 2024-06-05T13:24:49Z

ClickBench: Total hot run time: 30.45 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7583d95015619320f6746c19b06710fca42d1238, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.50	0.47	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.55	0.50	0.50
query10	0.55	0.55	0.55
query11	0.15	0.12	0.11
query12	0.14	0.12	0.12
query13	0.60	0.58	0.60
query14	0.78	0.79	0.79
query15	0.82	0.81	0.81
query16	0.36	0.36	0.37
query17	1.03	0.94	1.04
query18	0.23	0.26	0.25
query19	1.79	1.73	1.76
query20	0.02	0.01	0.00
query21	15.73	0.66	0.65
query22	4.43	7.23	1.88
query23	18.24	1.30	1.20
query24	2.07	0.22	0.21
query25	0.15	0.09	0.08
query26	0.27	0.17	0.17
query27	0.09	0.09	0.09
query28	13.26	1.01	1.00
query29	12.97	3.32	3.25
query30	0.24	0.05	0.05
query31	2.89	0.39	0.39
query32	3.27	0.47	0.46
query33	2.87	2.94	2.96
query34	17.07	4.43	4.41
query35	4.45	4.46	4.52
query36	0.65	0.46	0.46
query37	0.17	0.15	0.14
query38	0.14	0.13	0.14
query39	0.05	0.03	0.03
query40	0.16	0.16	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.13 s
Total hot run time: 30.45 s

feiniaofeiafei · 2024-06-06T03:37:14Z

run buildall

doris-robot · 2024-06-06T04:02:28Z

TPC-H: Total hot run time: 42095 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0265729bf20a0f7585e4b99ec838747cacb8b4e2, data reload: false

------ Round 1 ----------------------------------
q1	17621	4371	4274	4274
q2	2046	200	199	199
q3	10445	1294	1248	1248
q4	10187	926	825	825
q5	7475	2940	2688	2688
q6	234	139	143	139
q7	973	633	643	633
q8	9266	2171	2134	2134
q9	9235	6733	6790	6733
q10	9480	3904	3894	3894
q11	457	250	246	246
q12	460	259	255	255
q13	18550	3239	3237	3237
q14	276	235	236	235
q15	537	475	509	475
q16	535	401	403	401
q17	1005	660	625	625
q18	8599	7897	7772	7772
q19	7960	1422	1415	1415
q20	660	331	327	327
q21	5239	3996	4036	3996
q22	438	354	344	344
Total cold run time: 121678 ms
Total hot run time: 42095 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4616	4445	4410	4410
q2	378	298	262	262
q3	3214	3030	2801	2801
q4	1931	1645	1748	1645
q5	5495	5531	5547	5531
q6	230	130	132	130
q7	2238	1847	1868	1847
q8	3295	3400	3403	3400
q9	8660	8713	8742	8713
q10	4089	3688	3722	3688
q11	608	509	489	489
q12	812	626	644	626
q13	16215	3141	3176	3141
q14	314	283	263	263
q15	523	472	473	472
q16	511	433	428	428
q17	1860	1532	1542	1532
q18	7940	7570	7390	7390
q19	1697	1520	1619	1520
q20	2086	1788	1781	1781
q21	9101	4706	4692	4692
q22	654	578	541	541
Total cold run time: 76467 ms
Total hot run time: 55302 ms

doris-robot · 2024-06-06T04:13:21Z

TPC-DS: Total hot run time: 171565 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0265729bf20a0f7585e4b99ec838747cacb8b4e2, data reload: false

query1	930	380	386	380
query2	6456	2427	2409	2409
query3	6640	206	206	206
query4	19636	17497	17316	17316
query5	4129	457	457	457
query6	262	168	163	163
query7	4597	296	302	296
query8	327	290	282	282
query9	8416	2444	2415	2415
query10	448	309	281	281
query11	10405	10087	9998	9998
query12	135	95	86	86
query13	1645	368	368	368
query14	10055	6265	7535	6265
query15	237	188	181	181
query16	7811	266	267	266
query17	1383	526	508	508
query18	1944	273	267	267
query19	200	158	149	149
query20	89	84	83	83
query21	214	146	130	130
query22	4369	4015	3984	3984
query23	33821	33108	33097	33097
query24	10957	2827	2844	2827
query25	608	351	364	351
query26	1245	158	161	158
query27	3020	330	319	319
query28	7515	2047	2044	2044
query29	878	625	624	624
query30	279	146	150	146
query31	954	748	728	728
query32	94	52	55	52
query33	760	275	281	275
query34	947	479	461	461
query35	731	625	629	625
query36	1091	905	917	905
query37	156	69	73	69
query38	2870	2769	2734	2734
query39	885	780	771	771
query40	210	130	125	125
query41	61	53	51	51
query42	126	96	101	96
query43	584	577	558	558
query44	1176	721	739	721
query45	211	170	171	170
query46	1067	707	710	707
query47	1845	1757	1788	1757
query48	377	297	296	296
query49	1031	423	421	421
query50	785	392	398	392
query51	6760	6581	6648	6581
query52	108	92	92	92
query53	356	289	287	287
query54	857	455	451	451
query55	76	74	75	74
query56	277	266	261	261
query57	1148	1011	1058	1011
query58	261	257	255	255
query59	3506	3226	3163	3163
query60	304	274	278	274
query61	89	90	86	86
query62	654	470	461	461
query63	330	295	299	295
query64	8998	2227	1732	1732
query65	3221	3098	3113	3098
query66	803	333	329	329
query67	15388	15043	14719	14719
query68	4558	535	522	522
query69	514	407	330	330
query70	1124	1083	1105	1083
query71	394	266	274	266
query72	7281	5286	5593	5286
query73	743	320	320	320
query74	5799	5483	5478	5478
query75	3418	2682	2653	2653
query76	2440	922	891	891
query77	443	286	289	286
query78	10209	9892	9757	9757
query79	2182	519	506	506
query80	1006	473	467	467
query81	581	221	217	217
query82	700	105	103	103
query83	234	169	169	169
query84	244	89	83	83
query85	1844	278	264	264
query86	509	282	330	282
query87	3312	3094	3071	3071
query88	4258	2349	2338	2338
query89	488	395	381	381
query90	1789	190	193	190
query91	126	98	102	98
query92	63	51	54	51
query93	2235	581	510	510
query94	1220	188	191	188
query95	397	317	320	317
query96	594	274	272	272
query97	3180	3013	3012	3012
query98	218	212	202	202
query99	1365	837	833	833
Total cold run time: 271216 ms
Total hot run time: 171565 ms

doris-robot · 2024-06-06T04:18:52Z

ClickBench: Total hot run time: 30.32 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0265729bf20a0f7585e4b99ec838747cacb8b4e2, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.68	0.08	0.07
query5	0.50	0.49	0.50
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.54
query11	0.14	0.11	0.11
query12	0.15	0.12	0.11
query13	0.60	0.59	0.59
query14	0.81	0.77	0.79
query15	0.83	0.82	0.81
query16	0.36	0.35	0.35
query17	1.04	1.04	0.98
query18	0.22	0.24	0.27
query19	1.76	1.68	1.84
query20	0.01	0.01	0.01
query21	15.50	0.67	0.66
query22	4.15	6.91	1.81
query23	18.28	1.44	1.23
query24	2.10	0.22	0.21
query25	0.15	0.08	0.08
query26	0.27	0.17	0.18
query27	0.08	0.08	0.09
query28	13.21	1.01	0.99
query29	13.38	3.27	3.27
query30	0.24	0.05	0.05
query31	2.85	0.39	0.38
query32	3.28	0.47	0.46
query33	2.89	2.90	2.88
query34	17.09	4.40	4.40
query35	4.46	4.49	4.64
query36	0.64	0.46	0.47
query37	0.17	0.16	0.15
query38	0.14	0.13	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 110.01 s
Total hot run time: 30.32 s

feiniaofeiafei · 2024-06-06T06:10:56Z

run buildall

doris-robot · 2024-06-06T06:54:15Z

TPC-H: Total hot run time: 40678 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 58e39665e2b39c85ca939ce8a8e44cdc5a1168a7, data reload: false

------ Round 1 ----------------------------------
q1	17649	4286	4215	4215
q2	2031	181	184	181
q3	10461	1200	1121	1121
q4	10192	822	820	820
q5	7478	2710	2726	2710
q6	226	136	139	136
q7	965	603	620	603
q8	9221	2111	2077	2077
q9	9407	6672	6651	6651
q10	9313	3961	3910	3910
q11	457	240	244	240
q12	487	231	235	231
q13	17395	3216	3273	3216
q14	252	222	224	222
q15	516	489	478	478
q16	485	393	387	387
q17	981	572	643	572
q18	8406	7746	7713	7713
q19	6707	1262	1459	1262
q20	652	320	318	318
q21	5031	3279	4021	3279
q22	408	336	336	336
Total cold run time: 118720 ms
Total hot run time: 40678 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4578	4445	4390	4390
q2	378	273	274	273
q3	3152	2913	2944	2913
q4	1901	1587	1614	1587
q5	5387	5504	5481	5481
q6	223	126	129	126
q7	2216	1832	1800	1800
q8	3245	3374	3368	3368
q9	8642	8637	8662	8637
q10	4064	3826	3832	3826
q11	579	485	491	485
q12	776	609	585	585
q13	16137	3050	3122	3050
q14	312	258	263	258
q15	532	494	495	494
q16	464	429	427	427
q17	1784	1516	1506	1506
q18	7996	7734	7300	7300
q19	1752	1541	1452	1452
q20	3026	1786	1791	1786
q21	9315	4768	4638	4638
q22	630	523	550	523
Total cold run time: 77089 ms
Total hot run time: 54905 ms

feiniaofeiafei · 2024-10-12T09:48:07Z

run buildall

feiniaofeiafei · 2024-10-12T10:58:04Z

run cloud_p0

…42044) add transform: ```java /** * LogicalAggregate (groupByExpr=[c1#13], outputExpr=[c1#13, count(c1#13) AS `count(c1)`#15]) * +--LogicalUnion (outputs=[c1#13], regularChildrenOutputs=[[c1#9], [a#4], [a#7]]) * |--child1 (output = [[c1#9]]) * |--child2 (output = [[a#4]]) * +--child3 (output = [[a#7]]) * transform to: * LogicalAggregate (groupByExpr=[c1#13], outputExpr=[c1#13, sum0(count(c1)#19) AS `count(c1)`#15]) * +--LogicalUnion (outputs=[c1#13, count(c1)#19], regularChildrenOutputs=[[c1#9, count(c1)#16], * [a#4, count(a)#17], [a#7, count(a)#18]]) * |--LogicalAggregate (groupByExpr=[c1#9], outputExpr=[c1#9, count(c1#9) AS `count(c1)`#16]) * | +--child1 * |--LogicalAggregate (groupByExpr=[a#4], outputExpr=[a#4, count(a#4) AS `count(a)`#17]) * | +--child2 * +--LogicalAggregate (groupByExpr=[a#7], outputExpr=[a#7, count(a#7) AS `count(a)`#18]] * +--child3 */ ```

924060929 reviewed Apr 19, 2024

View reviewed changes

feiniaofeiafei force-pushed the union_sum0 branch from 03bec4d to 3eda8d1 Compare April 24, 2024 02:58

feiniaofeiafei force-pushed the union_sum0 branch from 3eda8d1 to 7583d95 Compare June 5, 2024 12:43

morrySnow added the not-merge/2.1 label Jun 6, 2024

feiniaofeiafei force-pushed the union_sum0 branch from 7583d95 to 0265729 Compare June 6, 2024 03:36

feiniaofeiafei force-pushed the union_sum0 branch from 0265729 to 58e3966 Compare June 6, 2024 06:10

feiniaofeiafei added 13 commits October 12, 2024 17:47

[Feat](nereids) add rewrite rule PushCountIntoUnionAll

276b6d5

remove useless print

f2dcadf

add regression and fix by comment

2cefcea

add regression test

4edbca0

[Feat](nereids) add rewrite rule PushCountIntoUnionAll

185fc95

add some feut

2fd336f

add some feut

2cde351

fix feut code style

7b0a85a

fix regression

9667747

fix ut

fbbdbe2

fix ut

eb9ceba

add logic for agg-project(1)-union

3f928a7

remove use less text

fa0fbcf

feiniaofeiafei force-pushed the union_sum0 branch from 4894539 to fa0fbcf Compare October 12, 2024 09:47

feiniaofeiafei requested review from 924060929 and morrySnow October 14, 2024 12:13

924060929 approved these changes Oct 15, 2024

View reviewed changes

morrySnow approved these changes Oct 15, 2024

View reviewed changes

morrySnow merged commit dc8c5a5 into apache:master Oct 15, 2024
25 of 27 checks passed

feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 17, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (apache#33530)

dc4db48

feiniaofeiafei mentioned this pull request Oct 17, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (#33530) #42044

Merged

feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 24, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (apache#33530)

65dd75a

feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 24, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (apache#33530)

df95704

feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 25, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (apache#33530)

73fd98c

feiniaofeiafei added a commit to feiniaofeiafei/doris that referenced this pull request Oct 25, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll (apache#33530)

4440ed4

924060929 added dev/3.0.3-merged and removed dev/3.0.x labels Oct 29, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](nereids) add rewrite rule PushCountIntoUnionAll #33530

[feature](nereids) add rewrite rule PushCountIntoUnionAll #33530

feiniaofeiafei commented Apr 11, 2024

doris-robot commented Apr 11, 2024

feiniaofeiafei commented Apr 11, 2024

feiniaofeiafei commented Apr 11, 2024

feiniaofeiafei commented Apr 15, 2024

feiniaofeiafei commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

924060929 Apr 19, 2024

feiniaofeiafei commented Apr 19, 2024

924060929 Apr 19, 2024

924060929 Apr 19, 2024 •

edited

Loading

feiniaofeiafei commented Apr 23, 2024

feiniaofeiafei commented Apr 23, 2024

feiniaofeiafei commented Apr 24, 2024

feiniaofeiafei commented Jun 5, 2024

doris-robot commented Jun 5, 2024

doris-robot commented Jun 5, 2024

doris-robot commented Jun 5, 2024

feiniaofeiafei commented Jun 6, 2024

doris-robot commented Jun 6, 2024

doris-robot commented Jun 6, 2024

doris-robot commented Jun 6, 2024

feiniaofeiafei commented Jun 6, 2024

doris-robot commented Jun 6, 2024

feiniaofeiafei commented Oct 12, 2024

feiniaofeiafei commented Oct 12, 2024

[feature](nereids) add rewrite rule PushCountIntoUnionAll #33530

[feature](nereids) add rewrite rule PushCountIntoUnionAll #33530

Conversation

feiniaofeiafei commented Apr 11, 2024

doris-robot commented Apr 11, 2024

feiniaofeiafei commented Apr 11, 2024

feiniaofeiafei commented Apr 11, 2024

feiniaofeiafei commented Apr 15, 2024

feiniaofeiafei commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

doris-robot commented Apr 15, 2024

924060929 Apr 19, 2024

Choose a reason for hiding this comment

feiniaofeiafei commented Apr 19, 2024

924060929 Apr 19, 2024

Choose a reason for hiding this comment

924060929 Apr 19, 2024 • edited Loading

Choose a reason for hiding this comment

feiniaofeiafei commented Apr 23, 2024

feiniaofeiafei commented Apr 23, 2024

feiniaofeiafei commented Apr 24, 2024

feiniaofeiafei commented Jun 5, 2024

doris-robot commented Jun 5, 2024

doris-robot commented Jun 5, 2024

doris-robot commented Jun 5, 2024

feiniaofeiafei commented Jun 6, 2024

doris-robot commented Jun 6, 2024

doris-robot commented Jun 6, 2024

doris-robot commented Jun 6, 2024

feiniaofeiafei commented Jun 6, 2024

doris-robot commented Jun 6, 2024

feiniaofeiafei commented Oct 12, 2024

feiniaofeiafei commented Oct 12, 2024

924060929 Apr 19, 2024 •

edited

Loading