Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids): Enhance LogicalJoin input validation when push down agg #37343

Merged
merged 1 commit into from
Jul 8, 2024
Merged

[fix](Nereids): Enhance LogicalJoin input validation when push down agg #37343

merged 1 commit into from
Jul 8, 2024

Conversation

keanji-x
Copy link
Contributor

@keanji-x keanji-x commented Jul 5, 2024 •
edited by morrySnow
Loading

intro by #36035

This PR refines the LogicalJoin class by introducing robust input validation. Key improvements:

  • Implement precise checks for join input validity
  • Ensure consistency between input slots and output sets
  • Gracefully handle various join scenarios (left/right)

These enhancements bolster query integrity and optimize join operations.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@keanji-x
Copy link
Contributor Author

keanji-x commented Jul 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40076 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f73f8ff7bc0ebaa8554769a83f25c9860b88f0bc, data reload: false

------ Round 1 ----------------------------------
q1	17597	4412	4274	4274
q2	2020	214	187	187
q3	10453	1209	1254	1209
q4	10202	812	784	784
q5	7508	2713	2678	2678
q6	216	136	136	136
q7	954	596	611	596
q8	9243	2111	2085	2085
q9	9048	6503	6476	6476
q10	9044	3749	3728	3728
q11	449	232	235	232
q12	463	224	225	224
q13	17780	2954	2987	2954
q14	269	219	219	219
q15	534	478	490	478
q16	499	376	365	365
q17	969	690	727	690
q18	8050	7564	7467	7467
q19	4220	1497	1552	1497
q20	654	328	316	316
q21	4901	3144	3911	3144
q22	374	339	337	337
Total cold run time: 115447 ms
Total hot run time: 40076 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4223	4220	4220
q2	361	260	270	260
q3	2978	2839	2883	2839
q4	1989	1728	1698	1698
q5	5744	5549	5490	5490
q6	238	132	131	131
q7	2187	1893	1895	1893
q8	3304	3469	3400	3400
q9	8769	8661	8811	8661
q10	4088	3919	3692	3692
q11	573	509	501	501
q12	808	663	668	663
q13	17267	3183	3162	3162
q14	324	269	281	269
q15	524	490	477	477
q16	481	443	449	443
q17	1812	1611	1519	1519
q18	8136	7944	7824	7824
q19	1788	1623	1645	1623
q20	2190	1842	1858	1842
q21	5275	4927	4896	4896
q22	602	547	575	547
Total cold run time: 73794 ms
Total hot run time: 56050 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f73f8ff7bc0ebaa8554769a83f25c9860b88f0bc, data reload: false

query1	909	367	367	367
query2	6463	2290	2300	2290
query3	6627	213	221	213
query4	28437	17272	17501	17272
query5	3678	491	505	491
query6	260	171	165	165
query7	4581	296	299	296
query8	322	277	272	272
query9	8534	2421	2400	2400
query10	595	290	289	289
query11	10688	9942	10084	9942
query12	120	89	83	83
query13	1660	384	368	368
query14	10259	7468	7138	7138
query15	237	180	180	180
query16	7756	295	303	295
query17	1367	536	533	533
query18	1959	274	285	274
query19	191	151	149	149
query20	91	79	79	79
query21	206	134	122	122
query22	4332	3946	3995	3946
query23	34005	34509	33918	33918
query24	11041	2888	2817	2817
query25	585	398	380	380
query26	716	156	160	156
query27	2308	317	321	317
query28	6138	2166	2161	2161
query29	874	640	617	617
query30	252	148	148	148
query31	992	786	748	748
query32	93	54	56	54
query33	698	298	306	298
query34	890	486	495	486
query35	771	647	637	637
query36	1126	968	985	968
query37	147	84	83	83
query38	2932	2838	2844	2838
query39	895	847	873	847
query40	213	130	166	130
query41	51	51	55	51
query42	116	100	98	98
query43	590	543	542	542
query44	1129	736	726	726
query45	192	155	158	155
query46	1073	740	708	708
query47	1836	1794	1750	1750
query48	369	306	298	298
query49	831	396	412	396
query50	769	387	380	380
query51	6799	7117	6749	6749
query52	110	91	96	91
query53	351	291	285	285
query54	876	443	437	437
query55	75	74	72	72
query56	282	255	263	255
query57	1166	1048	1057	1048
query58	259	242	256	242
query59	3383	3123	3040	3040
query60	310	299	276	276
query61	100	92	94	92
query62	567	466	433	433
query63	321	286	294	286
query64	9521	2175	1626	1626
query65	3190	3103	3129	3103
query66	677	318	325	318
query67	15378	15130	14830	14830
query68	4487	532	530	530
query69	438	366	314	314
query70	1179	1030	1135	1030
query71	381	284	281	281
query72	7008	5472	5345	5345
query73	727	327	321	321
query74	5991	5488	5443	5443
query75	3348	2693	2719	2693
query76	2507	878	926	878
query77	467	311	297	297
query78	9713	9313	8818	8818
query79	2693	515	536	515
query80	2045	489	479	479
query81	596	220	222	220
query82	789	112	103	103
query83	271	173	173	173
query84	281	83	94	83
query85	1952	316	302	302
query86	474	333	324	324
query87	3285	3087	3099	3087
query88	4536	2353	2377	2353
query89	472	375	375	375
query90	1677	189	189	189
query91	132	102	100	100
query92	62	50	52	50
query93	3741	520	508	508
query94	842	212	214	212
query95	415	318	319	318
query96	589	263	266	263
query97	3172	3009	3023	3009
query98	215	195	198	195
query99	1154	828	837	828
Total cold run time: 279908 ms
Total hot run time: 172690 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f73f8ff7bc0ebaa8554769a83f25c9860b88f0bc, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.09
query5	0.51	0.49	0.48
query6	1.13	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.47	0.50
query10	0.53	0.54	0.53
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.61	0.59	0.58
query14	0.75	0.78	0.78
query15	0.85	0.80	0.82
query16	0.37	0.35	0.37
query17	0.95	0.97	0.98
query18	0.22	0.25	0.26
query19	1.85	1.67	1.77
query20	0.01	0.01	0.01
query21	15.42	0.76	0.65
query22	4.12	7.84	1.91
query23	18.28	1.31	1.19
query24	1.92	0.27	0.23
query25	0.15	0.09	0.08
query26	0.30	0.21	0.21
query27	0.45	0.22	0.22
query28	13.24	1.02	0.99
query29	12.56	3.30	3.29
query30	0.25	0.06	0.05
query31	2.87	0.40	0.39
query32	3.26	0.48	0.47
query33	3.00	2.89	2.87
query34	17.48	4.41	4.31
query35	4.46	4.49	4.49
query36	0.65	0.46	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.04	0.04
query40	0.15	0.13	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.85 s
Total hot run time: 30.46 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@englefly englefly changed the title [fix](Nereids): Enhance LogicalJoin input validation when push dow agg [fix](Nereids): Enhance LogicalJoin input validation when push down agg Jul 8, 2024
@keanji-x keanji-x merged commit 9d7d7b9 into apache:master Jul 8, 2024
29 of 32 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@keanji-x @dataroaring
[fix](Nereids) Enhance LogicalJoin input validation when push down agg (
f32f44a 
#37343)

intro by #36035

This PR refines the LogicalJoin class by introducing robust input
validation. Key improvements:

* Implement precise checks for join input validity
* Ensure consistency between input slots and output sets
* Gracefully handle various join scenarios (left/right)

These enhancements bolster query integrity and optimize join operations.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@englefly englefly englefly approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged not-merge/2.1 p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@keanji-x @doris-robot @englefly @morrySnow @dataroaring