Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BugFix](Iceberg Catalog) Fix iceberg catalog of hms and hadoop not support iceberg properties #33113

Conversation

GoGoWen
Copy link
Contributor

@GoGoWen GoGoWen commented Apr 1, 2024

Proposed changes

when use use iceberg catalog with hms or hadoop type, we can not set the iceberg properties like "io.manifest.cache-enabled" to enable manifest cache in iceberg.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@GoGoWen
Copy link
Contributor Author

GoGoWen commented Apr 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39148 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a7d337ed45100cc628ec38f9bd87e62bd0ca9eea, data reload: false

------ Round 1 ----------------------------------
q1	17728	4222	4193	4193
q2	2641	207	194	194
q3	11171	1253	1428	1253
q4	11257	875	1015	875
q5	8101	3042	3015	3015
q6	226	138	136	136
q7	1136	673	618	618
q8	9611	2064	2085	2064
q9	6899	6218	6141	6141
q10	8437	3551	3499	3499
q11	424	243	237	237
q12	383	221	216	216
q13	17768	2918	2929	2918
q14	275	238	247	238
q15	527	482	499	482
q16	515	375	374	374
q17	982	932	929	929
q18	7430	6491	6572	6491
q19	1595	1548	1549	1548
q20	603	313	333	313
q21	3600	3112	3122	3112
q22	361	302	316	302
Total cold run time: 111670 ms
Total hot run time: 39148 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4076	4075	4064	4064
q2	331	229	223	223
q3	2984	2947	2935	2935
q4	1895	1837	1845	1837
q5	5278	5238	5246	5238
q6	209	124	123	123
q7	2258	1838	1803	1803
q8	3208	3279	3299	3279
q9	8556	8625	8474	8474
q10	3763	3870	3815	3815
q11	534	445	452	445
q12	714	545	527	527
q13	9455	2924	2954	2924
q14	292	272	249	249
q15	516	483	477	477
q16	461	416	396	396
q17	1742	1699	1662	1662
q18	7701	7355	7188	7188
q19	1652	1660	1647	1647
q20	1942	1734	1751	1734
q21	5039	4721	4827	4721
q22	502	423	452	423
Total cold run time: 63108 ms
Total hot run time: 54184 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 182591 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a7d337ed45100cc628ec38f9bd87e62bd0ca9eea, data reload: false

query1	1222	1134	1117	1117
query2	6496	1846	1808	1808
query3	6655	217	219	217
query4	24638	21396	21507	21396
query5	4176	434	413	413
query6	277	197	197	197
query7	4609	313	297	297
query8	241	184	188	184
query9	8490	2284	2265	2265
query10	588	265	256	256
query11	14994	14442	14446	14442
query12	140	96	91	91
query13	1639	382	377	377
query14	8743	6908	6810	6810
query15	218	180	186	180
query16	7055	286	283	283
query17	978	637	582	582
query18	1881	290	290	290
query19	204	165	164	164
query20	102	96	97	96
query21	201	131	125	125
query22	4958	4855	4880	4855
query23	33632	32902	32711	32711
query24	12509	3164	3136	3136
query25	703	398	413	398
query26	1921	166	167	166
query27	2990	331	349	331
query28	6729	1881	1860	1860
query29	1325	594	602	594
query30	316	155	154	154
query31	983	763	738	738
query32	101	61	64	61
query33	719	263	263	263
query34	1009	492	501	492
query35	837	703	701	701
query36	1006	880	850	850
query37	265	79	78	78
query38	3580	3391	3396	3391
query39	1580	1547	1548	1547
query40	304	137	136	136
query41	49	48	48	48
query42	109	106	104	104
query43	439	403	413	403
query44	1115	725	701	701
query45	282	251	266	251
query46	1057	800	787	787
query47	1884	1785	1780	1780
query48	380	314	306	306
query49	1166	365	368	365
query50	801	397	419	397
query51	6882	6825	6760	6760
query52	111	99	96	96
query53	367	294	292	292
query54	328	250	252	250
query55	98	82	80	80
query56	252	239	235	235
query57	1222	1144	1143	1143
query58	268	230	227	227
query59	2560	2411	2445	2411
query60	271	249	253	249
query61	114	113	111	111
query62	716	460	443	443
query63	309	281	283	281
query64	6450	3172	3176	3172
query65	3059	3044	3005	3005
query66	1445	328	327	327
query67	15629	15071	14883	14883
query68	9373	595	591	591
query69	623	328	321	321
query70	1410	1091	1084	1084
query71	526	286	282	282
query72	7090	2602	2444	2444
query73	1682	338	338	338
query74	6858	6425	6290	6290
query75	4173	2298	2286	2286
query76	6539	1099	1173	1099
query77	638	252	251	251
query78	10900	10309	10156	10156
query79	11801	529	533	529
query80	1624	433	419	419
query81	499	226	228	226
query82	398	107	101	101
query83	223	163	166	163
query84	274	90	91	90
query85	1166	305	292	292
query86	360	303	303	303
query87	3699	3468	3514	3468
query88	3767	2341	2349	2341
query89	544	372	366	366
query90	1990	175	184	175
query91	137	109	108	108
query92	63	52	56	52
query93	5456	544	531	531
query94	1331	198	197	197
query95	1098	1095	1091	1091
query96	607	277	272	272
query97	2641	2559	2506	2506
query98	227	217	220	217
query99	1155	863	843	843
Total cold run time: 305249 ms
Total hot run time: 182591 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a7d337ed45100cc628ec38f9bd87e62bd0ca9eea, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.04
query3	0.24	0.05	0.05
query4	1.67	0.07	0.06
query5	0.48	0.48	0.49
query6	1.13	0.66	0.65
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.57	0.52	0.52
query10	0.55	0.57	0.56
query11	0.14	0.10	0.11
query12	0.13	0.11	0.11
query13	0.61	0.59	0.59
query14	0.77	0.81	0.79
query15	0.87	0.83	0.84
query16	0.36	0.36	0.37
query17	0.98	0.97	0.98
query18	0.25	0.25	0.24
query19	1.84	1.76	1.73
query20	0.02	0.01	0.01
query21	15.42	0.75	0.66
query22	3.26	5.33	1.88
query23	17.81	1.22	1.08
query24	1.83	0.22	0.21
query25	0.12	0.09	0.09
query26	0.28	0.18	0.19
query27	0.09	0.09	0.08
query28	13.47	0.95	1.00
query29	12.60	3.29	3.45
query30	0.27	0.06	0.05
query31	2.85	0.39	0.39
query32	3.27	0.48	0.48
query33	2.83	2.82	2.95
query34	15.52	4.32	4.33
query35	4.35	4.33	4.34
query36	0.67	0.48	0.48
query37	0.20	0.16	0.17
query38	0.16	0.17	0.14
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.10	0.05	0.05
query42	0.06	0.06	0.05
query43	0.05	0.04	0.04
Total cold run time: 106.23 s
Total hot run time: 30.08 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit a7d337ed45100cc628ec38f9bd87e62bd0ca9eea with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       15.3 seconds inserted 10000000 Rows, about 653K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 2, 2024
Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Apr 2, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit b417ef3 into apache:master Apr 5, 2024
28 of 30 checks passed
yiguolei pushed a commit that referenced this pull request Apr 7, 2024
…upport iceberg properties (#33113)

* fix iceberg catalog of  hms and hadoop not support iceberg properties

* remove unused import
@yiguolei yiguolei mentioned this pull request Apr 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.x dev/2.1.2-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants