Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhencement](trino-connector) trino-connector supports push down projection to connectors #37874

Merged
merged 1 commit into from
Jul 25, 2024
Merged

[enhencement](trino-connector) trino-connector supports push down projection to connectors #37874

merged 1 commit into from
Jul 25, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Jul 16, 2024 •
edited
Loading

Invoke the applyProjection method of connectorMetadata` to push the projection down to the connector. This reduces the amount of data retrieved by the connector and enhances query performance.

Projection pushdown is particularly important for the BigQuery connector.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman morningman force-pushed the trinoConnectorSupportPrejectionPushDown branch from 81649d8 to bd733d8 Compare July 23, 2024 04:14
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39470 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bd733d8ce5ad498e467654034cfdeaf759383f27, data reload: false

------ Round 1 ----------------------------------
q1	17618	4341	4240	4240
q2	2017	187	189	187
q3	10460	1157	1074	1074
q4	10189	917	816	816
q5	7521	2666	2742	2666
q6	218	140	142	140
q7	953	597	595	595
q8	9221	2067	2050	2050
q9	8571	6540	6554	6540
q10	8848	3749	3730	3730
q11	459	238	238	238
q12	396	223	225	223
q13	17764	2963	2987	2963
q14	281	243	254	243
q15	532	476	464	464
q16	490	389	379	379
q17	954	637	674	637
q18	8124	7473	7367	7367
q19	6834	1429	1196	1196
q20	648	314	316	314
q21	4847	3129	3165	3129
q22	345	279	287	279
Total cold run time: 117290 ms
Total hot run time: 39470 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4251	4257	4251
q2	385	267	257	257
q3	2994	2889	2927	2889
q4	1976	1675	1631	1631
q5	5632	5482	5497	5482
q6	220	130	133	130
q7	2194	1872	1877	1872
q8	3319	3434	3422	3422
q9	8791	8796	8854	8796
q10	4093	3912	3747	3747
q11	619	521	501	501
q12	849	650	638	638
q13	15943	3166	3225	3166
q14	305	283	277	277
q15	535	492	477	477
q16	484	447	454	447
q17	1800	1508	1510	1508
q18	8084	7909	7996	7909
q19	1763	1633	1669	1633
q20	2149	1880	1842	1842
q21	5139	4938	4749	4749
q22	575	510	511	510
Total cold run time: 72214 ms
Total hot run time: 56134 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bd733d8ce5ad498e467654034cfdeaf759383f27, data reload: false

query1	905	380	361	361
query2	6435	2036	1889	1889
query3	6630	215	215	215
query4	28116	17632	17343	17343
query5	3666	470	474	470
query6	267	180	192	180
query7	4584	283	280	280
query8	254	185	185	185
query9	8361	2387	2357	2357
query10	430	287	269	269
query11	10636	10171	10150	10150
query12	123	85	90	85
query13	1676	359	352	352
query14	10001	8427	7655	7655
query15	230	165	168	165
query16	7736	496	472	472
query17	1381	543	520	520
query18	1965	270	294	270
query19	200	153	161	153
query20	92	83	82	82
query21	228	133	127	127
query22	4322	4093	3966	3966
query23	34284	33768	33812	33768
query24	10809	2932	2866	2866
query25	578	395	385	385
query26	713	154	146	146
query27	2257	275	290	275
query28	5915	2056	2049	2049
query29	889	636	630	630
query30	257	160	163	160
query31	963	772	783	772
query32	95	53	55	53
query33	654	342	363	342
query34	950	488	497	488
query35	928	744	752	744
query36	1150	984	953	953
query37	135	78	78	78
query38	2963	2773	2795	2773
query39	915	808	816	808
query40	204	121	119	119
query41	46	43	48	43
query42	122	99	99	99
query43	535	472	460	460
query44	1059	724	706	706
query45	197	164	161	161
query46	1073	745	714	714
query47	1827	1796	1784	1784
query48	353	303	312	303
query49	824	398	409	398
query50	770	389	405	389
query51	6776	6701	6657	6657
query52	105	90	97	90
query53	358	302	290	290
query54	874	443	443	443
query55	76	73	75	73
query56	306	283	286	283
query57	1172	1060	1060	1060
query58	267	250	286	250
query59	3002	2720	2768	2720
query60	343	274	271	271
query61	100	96	98	96
query62	805	641	641	641
query63	320	291	283	283
query64	9101	2204	1657	1657
query65	3179	3101	3118	3101
query66	748	330	368	330
query67	15739	15512	15211	15211
query68	6137	532	540	532
query69	675	461	367	367
query70	1194	1068	1064	1064
query71	447	281	273	273
query72	7423	5600	5396	5396
query73	776	327	321	321
query74	6135	5748	5700	5700
query75	3658	2695	2703	2695
query76	3229	953	862	862
query77	671	309	325	309
query78	9612	8997	9091	8997
query79	2400	522	508	508
query80	2582	469	462	462
query81	581	219	221	219
query82	1358	132	131	131
query83	276	164	195	164
query84	259	90	83	83
query85	1383	321	308	308
query86	473	297	309	297
query87	3248	3099	3109	3099
query88	3555	2342	2377	2342
query89	480	393	392	392
query90	1758	189	190	189
query91	128	99	99	99
query92	61	51	48	48
query93	2647	491	499	491
query94	1154	279	283	279
query95	402	322	316	316
query96	603	271	267	267
query97	3151	3044	3028	3028
query98	214	208	196	196
query99	1487	1257	1318	1257
Total cold run time: 281131 ms
Total hot run time: 174159 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.5 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bd733d8ce5ad498e467654034cfdeaf759383f27, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.50	0.51
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.50	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.13
query13	0.59	0.59	0.58
query14	0.75	0.78	0.81
query15	0.85	0.81	0.81
query16	0.34	0.37	0.36
query17	1.04	1.00	0.99
query18	0.22	0.21	0.22
query19	1.80	1.72	1.74
query20	0.02	0.01	0.01
query21	15.45	0.79	0.68
query22	4.20	8.03	1.69
query23	18.55	1.39	1.26
query24	2.12	0.24	0.24
query25	0.17	0.08	0.10
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.21	1.00	1.01
query29	12.60	3.30	3.28
query30	0.24	0.06	0.05
query31	2.86	0.38	0.38
query32	3.30	0.48	0.47
query33	2.82	2.97	2.91
query34	17.16	4.37	4.36
query35	4.43	4.41	4.40
query36	0.65	0.46	0.51
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.04	0.04
query43	0.05	0.03	0.03
Total cold run time: 109.93 s
Total hot run time: 30.5 s

morningman
morningman approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

hubgeter
hubgeter approved these changes Jul 25, 2024
View reviewed changes
Copy link
Contributor

@hubgeter hubgeter left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman morningman merged commit d6dd62e into apache:master Jul 25, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
@BePPPower @dataroaring
[enhencement](trino-connector) trino-connector supports push down pro…
9fbc89b 
…jection to connectors (#37874)

Invoke the `applyProjection` method of connectorMetadata` to push the
projection down to the connector. This reduces the amount of data
retrieved by the connector and enhances query performance.

Projection pushdown is particularly important for the BigQuery
connector.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hubgeter hubgeter hubgeter approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@BePPPower @doris-robot @morningman @hubgeter @dataroaring