Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nerieds) column prune should retain at least one column for union all (#41613) #41909

Merged
merged 1 commit into from
Oct 24, 2024
Merged

[fix](Nerieds) column prune should retain at least one column for union all (#41613) #41909

merged 1 commit into from
Oct 24, 2024

Conversation

feiniaofeiafei
Copy link
Contributor

cherry-pick #41613 to branch-3.0

@feiniaofeiafei
[fix](Nerieds) column prune should retain at least one column for uni…
3ce6d47 
…on all (apache#41613)


select count(1) from(select 3, 6 union all select 1, 3) t

wrong LogicalUnion plan:

LogicalUnion( qualifier=ALL, outputs=[3#6], regularChildrenOutputs=[], constantExprsList=[[], []], hasPushedFilter=false

this sql will report error in explain, because the logical union outputs
has a slot, but the logical union has no child and has a empty
constantExprList, which is wrong set in column prune.
this pr fixes it by consider when require columns is empty and keep the
min slot and min slot corresponding const expressions.
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@feiniaofeiafei
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40438 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ce6d472d283c6d3292d3739abd400b5e09f1d41, data reload: false

------ Round 1 ----------------------------------
q1	17758	7397	7351	7351
q2	2488	153	148	148
q3	11113	1125	1139	1125
q4	10231	756	687	687
q5	7735	2824	2743	2743
q6	226	147	145	145
q7	992	619	601	601
q8	9337	1900	1897	1897
q9	6617	6421	6421	6421
q10	6937	2315	2277	2277
q11	439	248	242	242
q12	414	209	210	209
q13	17787	2938	2994	2938
q14	249	207	202	202
q15	564	507	510	507
q16	479	405	400	400
q17	961	532	538	532
q18	7228	6720	6620	6620
q19	1358	931	936	931
q20	580	273	269	269
q21	3910	3194	3218	3194
q22	1120	1013	999	999
Total cold run time: 108523 ms
Total hot run time: 40438 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7244	7271	7161	7161
q2	330	225	230	225
q3	2865	2723	2694	2694
q4	1876	1621	1635	1621
q5	5328	5376	5366	5366
q6	221	136	136	136
q7	2088	1627	1639	1627
q8	3168	3335	3350	3335
q9	8496	8447	8445	8445
q10	3428	3396	3364	3364
q11	575	465	473	465
q12	749	567	590	567
q13	16902	2950	2969	2950
q14	283	250	255	250
q15	546	509	512	509
q16	488	438	427	427
q17	1778	1571	1556	1556
q18	7565	7476	7218	7218
q19	1629	1530	1531	1530
q20	1987	1792	1798	1792
q21	5188	4945	5064	4945
q22	1096	992	983	983
Total cold run time: 73830 ms
Total hot run time: 57166 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187524 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ce6d472d283c6d3292d3739abd400b5e09f1d41, data reload: false

query1	966	359	359	359
query2	6572	1984	1913	1913
query3	6697	207	231	207
query4	33762	23268	23325	23268
query5	4336	486	457	457
query6	268	173	176	173
query7	4612	299	305	299
query8	256	210	194	194
query9	9341	2643	2602	2602
query10	468	273	275	273
query11	18033	16102	15186	15186
query12	144	102	95	95
query13	1671	456	442	442
query14	9349	6765	6984	6765
query15	228	168	169	168
query16	7958	472	454	454
query17	1624	543	523	523
query18	2039	296	308	296
query19	232	140	136	136
query20	116	106	105	105
query21	199	99	96	96
query22	4353	4119	4143	4119
query23	34508	33815	33742	33742
query24	12195	2847	2767	2767
query25	712	369	378	369
query26	1880	150	157	150
query27	2957	285	284	284
query28	7918	2139	2113	2113
query29	1099	434	408	408
query30	313	146	140	140
query31	1049	786	787	786
query32	93	52	51	51
query33	759	278	285	278
query34	951	508	497	497
query35	871	713	721	713
query36	1112	905	932	905
query37	245	71	78	71
query38	3985	3918	3844	3844
query39	1474	1416	1420	1416
query40	296	94	93	93
query41	44	41	41	41
query42	116	93	98	93
query43	515	477	477	477
query44	1231	769	749	749
query45	190	163	164	163
query46	1130	700	692	692
query47	1890	1805	1845	1805
query48	421	341	329	329
query49	1241	372	370	370
query50	804	387	407	387
query51	6838	6908	6736	6736
query52	96	94	84	84
query53	258	188	178	178
query54	1245	448	448	448
query55	77	72	73	72
query56	279	240	247	240
query57	1220	1129	1116	1116
query58	256	240	232	232
query59	3033	2826	2750	2750
query60	296	263	254	254
query61	101	99	101	99
query62	870	669	660	660
query63	216	178	179	178
query64	5214	670	684	670
query65	3258	3158	3159	3158
query66	1423	320	304	304
query67	15750	15283	15215	15215
query68	3573	531	542	531
query69	431	291	295	291
query70	1161	1125	1136	1125
query71	332	275	266	266
query72	6227	3970	4110	3970
query73	781	331	333	331
query74	9851	8807	8848	8807
query75	3370	2624	2644	2624
query76	1978	828	845	828
query77	402	294	284	284
query78	9839	9138	9021	9021
query79	2929	578	571	571
query80	2102	465	460	460
query81	534	223	231	223
query82	851	130	127	127
query83	318	143	141	141
query84	300	84	88	84
query85	2023	287	279	279
query86	490	300	301	300
query87	4465	4219	4134	4134
query88	4644	2392	2387	2387
query89	428	282	280	280
query90	1939	189	179	179
query91	135	104	103	103
query92	64	47	46	46
query93	4845	538	525	525
query94	961	280	297	280
query95	341	240	242	240
query96	617	280	274	274
query97	3314	3071	3162	3071
query98	224	195	198	195
query99	1850	1290	1304	1290
Total cold run time: 304972 ms
Total hot run time: 187524 ms

@feiniaofeiafei
Copy link
Contributor Author

run p0

@feiniaofeiafei
Copy link
Contributor Author

run performance

@doris-robot
Copy link

TPC-H: Total hot run time: 40612 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3ce6d472d283c6d3292d3739abd400b5e09f1d41, data reload: false

------ Round 1 ----------------------------------
q1	18717	7424	7333	7333
q2	2054	163	150	150
q3	12732	1090	1143	1090
q4	10552	746	709	709
q5	7698	2836	2755	2755
q6	229	145	146	145
q7	1015	641	615	615
q8	10117	1912	1917	1912
q9	7963	6444	6492	6444
q10	7074	2372	2336	2336
q11	466	256	256	256
q12	418	221	221	221
q13	17951	3051	3094	3051
q14	263	223	220	220
q15	569	513	531	513
q16	531	430	427	427
q17	984	600	591	591
q18	8786	6648	6655	6648
q19	4858	1003	879	879
q20	584	268	280	268
q21	4014	3091	3195	3091
q22	1089	981	958	958
Total cold run time: 118664 ms
Total hot run time: 40612 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7414	7266	7350	7266
q2	326	236	247	236
q3	3004	2871	2891	2871
q4	2041	1739	1724	1724
q5	5657	5667	5754	5667
q6	229	140	140	140
q7	2213	1740	1843	1740
q8	3235	3438	3506	3438
q9	8811	8874	8741	8741
q10	3512	3499	3606	3499
q11	575	480	483	480
q12	814	600	637	600
q13	16688	3125	3123	3123
q14	305	270	280	270
q15	558	534	520	520
q16	505	479	475	475
q17	1832	1607	1605	1605
q18	8188	7689	7496	7496
q19	3390	1367	1535	1367
q20	2038	1849	1847	1847
q21	5432	5320	5194	5194
q22	1118	992	1016	992
Total cold run time: 77885 ms
Total hot run time: 59291 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 193374 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3ce6d472d283c6d3292d3739abd400b5e09f1d41, data reload: false

query1	1221	913	893	893
query2	6280	1941	1876	1876
query3	10815	3939	3817	3817
query4	67093	25984	23353	23353
query5	5665	494	458	458
query6	476	182	175	175
query7	6306	317	302	302
query8	309	213	198	198
query9	9654	2630	2645	2630
query10	518	284	274	274
query11	18413	15245	15954	15245
query12	156	102	100	100
query13	1648	479	451	451
query14	11428	7233	7322	7233
query15	222	188	175	175
query16	7224	490	463	463
query17	1070	566	545	545
query18	1672	298	301	298
query19	190	146	143	143
query20	113	110	108	108
query21	206	101	96	96
query22	4515	4220	4175	4175
query23	34206	34031	33718	33718
query24	5653	2956	2838	2838
query25	507	404	397	397
query26	674	159	161	159
query27	1707	295	311	295
query28	3718	2159	2154	2154
query29	660	433	454	433
query30	228	148	158	148
query31	984	756	806	756
query32	69	52	56	52
query33	445	288	295	288
query34	910	495	508	495
query35	827	734	717	717
query36	1097	942	939	939
query37	128	77	76	76
query38	4007	3933	3918	3918
query39	1474	1416	1453	1416
query40	209	97	99	97
query41	45	42	42	42
query42	121	98	98	98
query43	528	465	482	465
query44	1109	760	767	760
query45	201	171	167	167
query46	1123	750	708	708
query47	1925	1812	1832	1812
query48	429	349	347	347
query49	729	395	395	395
query50	816	423	414	414
query51	6900	6934	6794	6794
query52	101	89	87	87
query53	270	187	190	187
query54	598	458	470	458
query55	71	75	73	73
query56	252	240	242	240
query57	1187	1143	1144	1143
query58	212	225	227	225
query59	2982	2963	3249	2963
query60	285	258	270	258
query61	102	102	98	98
query62	761	672	654	654
query63	212	188	185	185
query64	1488	629	605	605
query65	3201	3162	3191	3162
query66	771	301	306	301
query67	15918	15365	15455	15365
query68	3621	536	542	536
query69	636	292	288	288
query70	1108	1140	1111	1111
query71	420	270	301	270
query72	7763	3881	3895	3881
query73	774	335	334	334
query74	10229	8843	9101	8843
query75	3746	2689	2630	2630
query76	3007	860	939	860
query77	761	287	284	284
query78	9991	9353	9154	9154
query79	5515	577	574	574
query80	1205	431	429	429
query81	569	218	234	218
query82	813	124	123	123
query83	388	143	142	142
query84	290	78	80	78
query85	1724	289	287	287
query86	468	298	305	298
query87	4595	4256	4306	4256
query88	4682	2359	2398	2359
query89	409	288	291	288
query90	2048	183	184	183
query91	139	104	112	104
query92	60	45	44	44
query93	5858	526	525	525
query94	1017	288	251	251
query95	357	246	247	246
query96	621	282	275	275
query97	3329	3105	3097	3097
query98	229	204	200	200
query99	1554	1296	1301	1296
Total cold run time: 332960 ms
Total hot run time: 193374 ms

@924060929 924060929 merged commit 12eb9c3 into apache:branch-3.0 Oct 24, 2024
22 of 24 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@feiniaofeiafei @doris-robot @924060929