Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](iceberg) iceberg write support insert overwrite and optimize hive write transaction statistics and #37191

Merged
merged 4 commits into from
Jul 18, 2024

Conversation

ghkang98
Copy link
Contributor

@ghkang98 ghkang98 commented Jul 3, 2024

iceberg support insert overwrite

  1. Iceberg transaction adds insert overwrite feature
    2.Iceberg transaction merge the commit functions of table and partition
    3.Use class SimpleTableInfo to replace the internal class DatabaseTableName of HmsTransaction
    4.Unified the file statistics of hive and iceberg write

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ghkang98
Copy link
Contributor Author

ghkang98 commented Jul 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40262 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e86531ec5ab3544bbfa0d9bf623d2b1686332cce, data reload: false

------ Round 1 ----------------------------------
q1	17631	4411	4346	4346
q2	2018	197	210	197
q3	10453	1176	1121	1121
q4	10184	808	794	794
q5	7511	2640	2625	2625
q6	229	145	142	142
q7	972	613	613	613
q8	9246	2105	2104	2104
q9	8970	6543	6518	6518
q10	9022	3813	3742	3742
q11	460	235	245	235
q12	434	240	235	235
q13	17773	3007	3005	3005
q14	260	237	221	221
q15	510	477	478	477
q16	487	372	374	372
q17	969	753	657	657
q18	8041	7513	7412	7412
q19	7354	1507	1512	1507
q20	651	321	329	321
q21	4967	3297	3279	3279
q22	388	347	339	339
Total cold run time: 118530 ms
Total hot run time: 40262 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4536	4383	4286	4286
q2	371	281	266	266
q3	3058	2919	3006	2919
q4	2071	1793	1828	1793
q5	5638	5560	5469	5469
q6	229	152	138	138
q7	2278	1915	1897	1897
q8	3325	3475	3429	3429
q9	8780	8835	8809	8809
q10	4175	3816	3820	3816
q11	625	511	495	495
q12	853	663	641	641
q13	17313	3363	3425	3363
q14	327	316	311	311
q15	545	500	497	497
q16	514	445	458	445
q17	1878	1559	1541	1541
q18	9038	8698	8056	8056
q19	1792	1581	1603	1581
q20	2130	1878	1972	1878
q21	8706	5025	4992	4992
q22	675	583	587	583
Total cold run time: 78857 ms
Total hot run time: 57205 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174316 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e86531ec5ab3544bbfa0d9bf623d2b1686332cce, data reload: false

query1	924	383	367	367
query2	6462	2577	2372	2372
query3	6636	202	206	202
query4	19360	17297	17310	17297
query5	3630	470	468	468
query6	269	161	181	161
query7	4600	312	298	298
query8	315	311	309	309
query9	8486	2388	2361	2361
query10	595	298	288	288
query11	10565	10215	10014	10014
query12	117	88	82	82
query13	1662	381	378	378
query14	10406	6672	7829	6672
query15	260	180	190	180
query16	8047	286	286	286
query17	1874	562	543	543
query18	2085	293	290	290
query19	341	161	163	161
query20	93	86	86	86
query21	220	137	126	126
query22	4534	4211	4012	4012
query23	34352	33511	33603	33511
query24	10639	2911	2933	2911
query25	627	404	419	404
query26	1022	167	169	167
query27	2197	329	325	325
query28	6128	2179	2172	2172
query29	938	674	670	670
query30	243	161	166	161
query31	972	779	767	767
query32	101	65	55	55
query33	770	322	308	308
query34	1020	487	515	487
query35	738	649	643	643
query36	1149	985	1016	985
query37	149	78	82	78
query38	2939	2889	2882	2882
query39	910	832	830	830
query40	222	138	134	134
query41	58	57	57	57
query42	120	110	103	103
query43	577	541	536	536
query44	1260	740	742	740
query45	197	166	178	166
query46	1085	747	724	724
query47	1862	1759	1771	1759
query48	369	309	308	308
query49	959	402	407	402
query50	775	395	393	393
query51	6872	6860	6738	6738
query52	108	89	90	89
query53	369	308	303	303
query54	932	450	452	450
query55	76	73	75	73
query56	278	261	260	260
query57	1105	1066	1068	1066
query58	247	253	250	250
query59	3598	3140	3179	3140
query60	307	280	276	276
query61	116	90	88	88
query62	600	441	447	441
query63	319	297	295	295
query64	8825	2253	1743	1743
query65	3148	3136	3122	3122
query66	746	326	334	326
query67	15546	15038	15005	15005
query68	8446	545	551	545
query69	740	426	345	345
query70	1258	1109	1159	1109
query71	518	296	288	288
query72	8398	5388	5660	5388
query73	1870	342	323	323
query74	5975	5536	5526	5526
query75	5232	2690	2702	2690
query76	5167	949	976	949
query77	766	309	308	308
query78	10404	9823	9887	9823
query79	6902	514	558	514
query80	943	472	491	472
query81	553	222	217	217
query82	767	110	104	104
query83	338	173	171	171
query84	270	89	86	86
query85	1284	269	264	264
query86	404	337	308	308
query87	3274	3119	3098	3098
query88	4169	2448	2481	2448
query89	502	390	385	385
query90	1971	188	189	188
query91	125	100	142	100
query92	57	49	51	49
query93	5281	519	514	514
query94	1173	192	187	187
query95	409	317	318	317
query96	610	270	264	264
query97	3159	3019	3048	3019
query98	228	199	191	191
query99	1127	853	830	830
Total cold run time: 289371 ms
Total hot run time: 174316 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.92 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e86531ec5ab3544bbfa0d9bf623d2b1686332cce, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.07	0.06
query5	0.49	0.48	0.48
query6	1.13	0.74	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.48	0.48
query10	0.56	0.54	0.54
query11	0.15	0.12	0.11
query12	0.14	0.13	0.13
query13	0.59	0.58	0.58
query14	0.78	0.78	0.78
query15	0.85	0.82	0.81
query16	0.38	0.37	0.37
query17	0.97	0.98	1.04
query18	0.22	0.27	0.23
query19	1.79	1.72	1.71
query20	0.02	0.01	0.01
query21	15.43	0.76	0.66
query22	3.44	7.61	2.43
query23	18.34	1.41	1.21
query24	2.01	0.27	0.22
query25	0.14	0.10	0.09
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.20	1.03	1.01
query29	12.58	3.30	3.20
query30	0.25	0.06	0.07
query31	2.86	0.40	0.39
query32	3.28	0.48	0.47
query33	2.91	2.93	2.85
query34	17.14	4.37	4.41
query35	4.50	4.50	4.53
query36	0.66	0.46	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.17	0.17	0.15
query41	0.09	0.04	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.55 s
Total hot run time: 30.92 s

@ghkang98
Copy link
Contributor Author

ghkang98 commented Jul 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40096 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35427cab63c118ea58e35f7b56d6a9de327b488c, data reload: false

------ Round 1 ----------------------------------
q1	17737	4382	4317	4317
q2	2019	190	200	190
q3	10445	1152	1075	1075
q4	10300	790	779	779
q5	8917	2641	2653	2641
q6	222	137	139	137
q7	966	586	608	586
q8	9206	2087	2052	2052
q9	9189	6538	6613	6538
q10	9212	3866	3806	3806
q11	511	250	243	243
q12	467	249	232	232
q13	17939	3198	3136	3136
q14	261	234	237	234
q15	516	503	496	496
q16	949	435	428	428
q17	1341	635	699	635
q18	8590	7584	7414	7414
q19	8571	1573	1386	1386
q20	649	321	328	321
q21	4820	3109	3869	3109
q22	387	341	349	341
Total cold run time: 123214 ms
Total hot run time: 40096 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4368	4204	4330	4204
q2	383	277	267	267
q3	3135	2894	2896	2894
q4	1998	1726	1648	1648
q5	5501	5437	5500	5437
q6	228	139	133	133
q7	2169	1843	1833	1833
q8	3249	3394	3711	3394
q9	8652	8777	8632	8632
q10	3975	3746	3830	3746
q11	592	512	508	508
q12	811	651	643	643
q13	16293	3186	3157	3157
q14	303	282	276	276
q15	523	485	483	483
q16	469	426	417	417
q17	1795	1539	1495	1495
q18	8108	7845	7656	7656
q19	1761	1596	1672	1596
q20	2210	1858	1883	1858
q21	5552	4786	4952	4786
q22	654	548	590	548
Total cold run time: 72729 ms
Total hot run time: 55611 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171570 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35427cab63c118ea58e35f7b56d6a9de327b488c, data reload: false

query1	912	382	370	370
query2	6425	2443	2353	2353
query3	6632	214	215	214
query4	21974	17696	17202	17202
query5	3706	477	465	465
query6	256	173	172	172
query7	4590	301	289	289
query8	325	288	295	288
query9	8664	2397	2392	2392
query10	604	303	285	285
query11	10437	9966	10004	9966
query12	119	84	81	81
query13	1661	370	356	356
query14	10164	7605	6686	6686
query15	246	182	182	182
query16	8003	304	305	304
query17	1834	544	516	516
query18	2046	264	271	264
query19	197	155	152	152
query20	93	90	81	81
query21	212	130	128	128
query22	4140	3941	4046	3941
query23	33797	33490	33616	33490
query24	10652	2834	2895	2834
query25	601	381	408	381
query26	739	156	153	153
query27	2297	338	324	324
query28	6044	2160	2136	2136
query29	882	636	629	629
query30	249	160	156	156
query31	980	752	758	752
query32	102	50	59	50
query33	663	294	283	283
query34	906	471	481	471
query35	761	633	628	628
query36	1151	948	986	948
query37	139	91	79	79
query38	2974	2849	2881	2849
query39	889	847	825	825
query40	205	125	127	125
query41	53	52	52	52
query42	103	100	98	98
query43	589	559	558	558
query44	1069	733	721	721
query45	202	165	167	165
query46	1064	731	708	708
query47	1899	1816	1763	1763
query48	365	299	302	299
query49	848	404	406	404
query50	762	372	385	372
query51	6831	6700	6750	6700
query52	97	92	87	87
query53	368	285	280	280
query54	853	435	436	435
query55	73	71	71	71
query56	277	263	269	263
query57	1159	1057	1047	1047
query58	271	252	260	252
query59	3328	3408	3150	3150
query60	346	288	280	280
query61	95	89	88	88
query62	615	455	450	450
query63	316	288	286	286
query64	8541	2218	1725	1725
query65	3149	3109	3087	3087
query66	742	322	323	322
query67	15823	14967	14847	14847
query68	4509	533	533	533
query69	478	315	321	315
query70	1122	1089	1071	1071
query71	378	286	281	281
query72	7225	5674	4977	4977
query73	741	329	328	328
query74	5998	5578	5479	5479
query75	3350	2656	2688	2656
query76	2383	897	996	897
query77	421	302	294	294
query78	9483	8940	8840	8840
query79	3549	510	519	510
query80	2203	534	462	462
query81	588	220	222	220
query82	1008	109	107	107
query83	279	176	173	173
query84	280	89	86	86
query85	1996	277	268	268
query86	497	311	298	298
query87	3323	3099	3106	3099
query88	4139	2365	2358	2358
query89	484	375	381	375
query90	1825	182	186	182
query91	127	99	99	99
query92	59	51	48	48
query93	5428	496	501	496
query94	1205	205	207	205
query95	408	313	310	310
query96	608	267	266	266
query97	3209	3079	3009	3009
query98	214	198	193	193
query99	1191	836	825	825
Total cold run time: 275817 ms
Total hot run time: 171570 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35427cab63c118ea58e35f7b56d6a9de327b488c, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.08	0.08
query5	0.50	0.48	0.50
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.54	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.11	0.12
query13	0.59	0.59	0.58
query14	0.78	0.78	0.79
query15	0.87	0.84	0.83
query16	0.37	0.37	0.37
query17	1.00	1.04	1.01
query18	0.23	0.23	0.24
query19	1.91	1.72	1.69
query20	0.01	0.02	0.01
query21	15.40	0.74	0.66
query22	4.09	6.63	2.35
query23	18.26	1.41	1.31
query24	2.11	0.24	0.21
query25	0.16	0.09	0.08
query26	0.30	0.21	0.22
query27	0.45	0.24	0.23
query28	13.28	1.04	1.03
query29	12.64	3.40	3.39
query30	0.26	0.06	0.05
query31	2.96	0.39	0.41
query32	3.22	0.49	0.47
query33	2.91	2.98	2.95
query34	17.10	4.35	4.30
query35	4.39	4.46	4.40
query36	0.67	0.50	0.49
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.77 s
Total hot run time: 31.28 s

@ghkang98
Copy link
Contributor Author

ghkang98 commented Jul 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39694 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35427cab63c118ea58e35f7b56d6a9de327b488c, data reload: false

------ Round 1 ----------------------------------
q1	17620	4385	4323	4323
q2	2015	188	200	188
q3	10445	1176	1138	1138
q4	10203	807	819	807
q5	7502	2660	2601	2601
q6	217	138	137	137
q7	949	597	610	597
q8	9223	2030	2061	2030
q9	8873	6443	6514	6443
q10	8957	3661	3733	3661
q11	445	239	238	238
q12	439	228	227	227
q13	18860	2988	2999	2988
q14	262	227	232	227
q15	518	497	473	473
q16	494	378	380	378
q17	945	614	674	614
q18	7956	7461	7345	7345
q19	7453	1540	1401	1401
q20	661	327	328	327
q21	4889	3214	3939	3214
q22	387	337	338	337
Total cold run time: 119313 ms
Total hot run time: 39694 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4395	4243	4232	4232
q2	372	278	267	267
q3	3065	2872	2923	2872
q4	1967	1716	1724	1716
q5	5552	5476	5442	5442
q6	218	132	142	132
q7	2226	1868	1855	1855
q8	3259	3405	3421	3405
q9	8630	8694	8758	8694
q10	4119	4009	3727	3727
q11	599	480	499	480
q12	827	626	613	613
q13	17191	3159	3201	3159
q14	312	286	282	282
q15	545	473	486	473
q16	491	441	431	431
q17	1785	1546	1486	1486
q18	8128	7971	7904	7904
q19	1803	1637	1657	1637
q20	3068	1893	1851	1851
q21	5225	4856	4704	4704
q22	645	532	567	532
Total cold run time: 74422 ms
Total hot run time: 55894 ms

@ghkang98 ghkang98 force-pushed the iceberg-write-v1 branch from 35427ca to ecb3f90 Compare July 8, 2024 15:33
@ghkang98
Copy link
Contributor Author

ghkang98 commented Jul 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39528 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ecb3f90b28c4110b1ab07dc73e130d3b67db3501, data reload: false

------ Round 1 ----------------------------------
q1	17611	4310	4251	4251
q2	2021	190	185	185
q3	10466	1189	1094	1094
q4	10196	768	826	768
q5	7528	2672	2614	2614
q6	220	135	134	134
q7	943	598	593	593
q8	9237	2091	2037	2037
q9	8851	6504	6457	6457
q10	8904	3748	3798	3748
q11	461	242	238	238
q12	443	231	234	231
q13	18817	2957	2986	2957
q14	271	219	229	219
q15	540	491	482	482
q16	504	372	368	368
q17	955	663	690	663
q18	7891	7381	7352	7352
q19	7704	1580	1349	1349
q20	674	313	330	313
q21	4809	3137	4009	3137
q22	396	338	346	338
Total cold run time: 119442 ms
Total hot run time: 39528 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4423	4815	4238	4238
q2	357	271	259	259
q3	3100	2887	2892	2887
q4	1950	1670	1748	1670
q5	5553	5464	5398	5398
q6	221	132	129	129
q7	2223	1878	1825	1825
q8	3262	3408	3406	3406
q9	8657	8705	8704	8704
q10	4150	3776	3774	3774
q11	588	504	496	496
q12	830	663	641	641
q13	17336	3151	3209	3151
q14	302	300	280	280
q15	541	494	473	473
q16	493	426	433	426
q17	1803	1509	1503	1503
q18	7998	7857	7674	7674
q19	1806	1555	1655	1555
q20	2187	1888	1826	1826
q21	5110	5031	4793	4793
q22	648	551	563	551
Total cold run time: 73538 ms
Total hot run time: 55659 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173418 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ecb3f90b28c4110b1ab07dc73e130d3b67db3501, data reload: false

query1	928	377	366	366
query2	6457	2408	2303	2303
query3	6630	208	217	208
query4	26093	17498	17332	17332
query5	3776	490	518	490
query6	273	184	176	176
query7	4588	296	285	285
query8	361	299	286	286
query9	8506	2388	2381	2381
query10	566	283	284	283
query11	11819	10181	10161	10161
query12	126	84	81	81
query13	1633	368	390	368
query14	10142	7350	7762	7350
query15	230	194	186	186
query16	7735	303	306	303
query17	1664	540	518	518
query18	1913	277	274	274
query19	196	151	146	146
query20	84	84	81	81
query21	208	131	121	121
query22	4551	3964	3922	3922
query23	33982	33704	33394	33394
query24	11353	2870	2955	2870
query25	574	397	395	395
query26	1037	152	147	147
query27	2317	277	281	277
query28	6784	2133	2112	2112
query29	872	646	628	628
query30	257	165	167	165
query31	964	743	783	743
query32	97	54	59	54
query33	755	297	304	297
query34	913	505	506	505
query35	711	597	602	597
query36	1125	948	974	948
query37	143	88	81	81
query38	2926	2799	2738	2738
query39	865	823	801	801
query40	211	124	129	124
query41	55	53	51	51
query42	111	100	99	99
query43	576	537	535	535
query44	1209	729	731	729
query45	195	157	163	157
query46	1067	730	712	712
query47	1886	1771	1818	1771
query48	362	291	291	291
query49	855	407	405	405
query50	777	394	403	394
query51	6825	6765	6801	6765
query52	105	93	92	92
query53	363	292	285	285
query54	940	442	438	438
query55	77	74	76	74
query56	291	265	260	260
query57	1121	1059	1070	1059
query58	251	248	247	247
query59	3381	3247	3087	3087
query60	312	291	271	271
query61	95	95	98	95
query62	790	636	657	636
query63	328	284	280	280
query64	9333	2197	1640	1640
query65	3250	3098	3129	3098
query66	713	332	339	332
query67	15288	14778	15019	14778
query68	4469	539	532	532
query69	541	479	349	349
query70	1168	1097	1156	1097
query71	387	293	291	291
query72	7165	5348	5872	5348
query73	745	325	424	325
query74	5882	5541	5598	5541
query75	3334	2690	2681	2681
query76	2392	867	949	867
query77	454	309	306	306
query78	9724	8970	8930	8930
query79	6408	502	504	502
query80	2313	471	476	471
query81	597	218	217	217
query82	1544	140	133	133
query83	279	167	167	167
query84	279	84	85	84
query85	1927	395	308	308
query86	489	321	324	321
query87	3319	3126	3084	3084
query88	5182	2353	2338	2338
query89	484	390	371	371
query90	1851	187	188	187
query91	128	104	100	100
query92	67	47	49	47
query93	5468	511	497	497
query94	1189	205	210	205
query95	401	316	319	316
query96	607	265	260	260
query97	3224	3043	3020	3020
query98	225	203	215	203
query99	1623	1242	1252	1242
Total cold run time: 288786 ms
Total hot run time: 173418 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.11 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ecb3f90b28c4110b1ab07dc73e130d3b67db3501, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.66	0.10	0.09
query5	0.50	0.49	0.48
query6	1.16	0.74	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.56	0.56	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.79	0.78
query15	0.87	0.82	0.82
query16	0.38	0.36	0.38
query17	1.04	0.96	1.05
query18	0.23	0.22	0.21
query19	1.92	1.73	1.69
query20	0.01	0.01	0.01
query21	15.39	0.74	0.65
query22	3.92	7.65	2.37
query23	18.27	1.41	1.23
query24	2.19	0.22	0.22
query25	0.15	0.09	0.09
query26	0.28	0.21	0.22
query27	0.45	0.23	0.24
query28	13.20	1.01	1.02
query29	12.66	3.29	3.26
query30	0.25	0.06	0.05
query31	2.87	0.39	0.38
query32	3.31	0.48	0.47
query33	2.96	2.95	2.95
query34	16.97	4.36	4.38
query35	4.42	4.44	4.50
query36	0.65	0.47	0.48
query37	0.18	0.15	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.62 s
Total hot run time: 31.11 s

@ghkang98
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40190 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0116e835e9e40476b5bba9ff0d10f2640f6b12e7, data reload: false

------ Round 1 ----------------------------------
q1	18169	4520	4383	4383
q2	2055	195	189	189
q3	10457	1208	1079	1079
q4	10191	837	711	711
q5	7550	2732	2682	2682
q6	221	137	139	137
q7	961	600	613	600
q8	9222	2096	2111	2096
q9	8725	6588	6572	6572
q10	8759	3832	3852	3832
q11	462	241	241	241
q12	404	225	228	225
q13	17769	3003	2998	2998
q14	268	237	230	230
q15	527	473	492	473
q16	502	391	383	383
q17	986	614	653	614
q18	8179	7651	7562	7562
q19	8407	1462	1369	1369
q20	713	335	356	335
q21	4982	3195	3332	3195
q22	347	286	284	284
Total cold run time: 119856 ms
Total hot run time: 40190 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4262	4215	4215
q2	370	285	275	275
q3	3024	2764	2767	2764
q4	1889	1609	1648	1609
q5	5308	5366	5326	5326
q6	220	132	132	132
q7	2164	1737	1741	1737
q8	3198	3340	3302	3302
q9	8454	8437	8401	8401
q10	3909	3662	3691	3662
q11	599	491	487	487
q12	778	604	600	600
q13	17586	2970	2970	2970
q14	295	277	269	269
q15	507	491	474	474
q16	487	410	431	410
q17	1788	1477	1508	1477
q18	7646	7501	7336	7336
q19	1661	1572	1611	1572
q20	1971	1795	1777	1777
q21	4859	4706	4832	4706
q22	592	523	507	507
Total cold run time: 71692 ms
Total hot run time: 54008 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173170 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0116e835e9e40476b5bba9ff0d10f2640f6b12e7, data reload: false

query1	918	389	380	380
query2	6461	1893	1827	1827
query3	6663	209	216	209
query4	23778	17444	17344	17344
query5	4178	480	488	480
query6	297	176	173	173
query7	4595	313	292	292
query8	255	206	195	195
query9	8483	2458	2442	2442
query10	450	275	267	267
query11	11858	9921	10001	9921
query12	131	85	84	84
query13	1634	370	378	370
query14	8672	7866	7552	7552
query15	217	170	165	165
query16	7635	322	320	320
query17	1801	559	542	542
query18	1899	298	286	286
query19	233	148	151	148
query20	91	80	83	80
query21	206	123	126	123
query22	4406	4245	4061	4061
query23	34139	33849	33260	33260
query24	11810	2939	2822	2822
query25	575	394	365	365
query26	1381	146	146	146
query27	3025	275	271	271
query28	7795	2045	2015	2015
query29	890	619	603	603
query30	281	149	147	147
query31	965	758	738	738
query32	98	55	55	55
query33	769	298	292	292
query34	1022	479	482	479
query35	707	572	572	572
query36	1103	945	962	945
query37	237	80	78	78
query38	2856	2728	2723	2723
query39	877	813	809	809
query40	283	119	121	119
query41	48	44	44	44
query42	118	100	101	100
query43	507	446	475	446
query44	1227	751	736	736
query45	190	166	157	157
query46	1084	743	723	723
query47	1826	1762	1798	1762
query48	370	287	289	287
query49	1183	410	403	403
query50	785	400	405	400
query51	6888	6772	6896	6772
query52	107	94	103	94
query53	354	295	300	295
query54	961	446	445	445
query55	76	75	76	75
query56	325	262	270	262
query57	1117	1053	1045	1045
query58	254	246	241	241
query59	2846	2541	2639	2541
query60	303	282	291	282
query61	99	94	92	92
query62	839	626	657	626
query63	320	289	291	289
query64	10439	2224	1725	1725
query65	3176	3150	3101	3101
query66	1348	338	330	330
query67	15615	14979	14984	14979
query68	4526	549	549	549
query69	478	317	334	317
query70	1176	1165	1173	1165
query71	373	281	291	281
query72	7039	5836	5848	5836
query73	749	324	326	324
query74	6042	5660	5670	5660
query75	3491	2665	2660	2660
query76	2831	948	972	948
query77	508	310	309	309
query78	9531	8999	9036	8999
query79	2183	525	529	525
query80	2191	484	473	473
query81	589	225	222	222
query82	763	136	128	128
query83	281	173	169	169
query84	270	94	88	88
query85	1491	322	307	307
query86	473	309	326	309
query87	3312	3076	3070	3070
query88	4102	2375	2393	2375
query89	496	397	372	372
query90	1934	194	198	194
query91	131	101	102	101
query92	60	50	51	50
query93	2051	525	513	513
query94	1291	214	280	214
query95	411	322	321	321
query96	586	278	269	269
query97	3221	2993	3023	2993
query98	215	203	198	198
query99	1744	1279	1289	1279
Total cold run time: 280618 ms
Total hot run time: 173170 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.08 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0116e835e9e40476b5bba9ff0d10f2640f6b12e7, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.47	0.48	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.76	0.81	0.77
query15	0.85	0.82	0.80
query16	0.36	0.37	0.36
query17	1.00	1.04	0.96
query18	0.22	0.21	0.21
query19	1.80	1.71	1.66
query20	0.01	0.01	0.01
query21	15.40	0.73	0.65
query22	4.03	6.30	2.48
query23	18.31	1.32	1.23
query24	2.09	0.24	0.23
query25	0.15	0.09	0.08
query26	0.31	0.22	0.21
query27	0.45	0.24	0.23
query28	13.22	1.01	1.00
query29	12.64	3.36	3.36
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.27	0.48	0.46
query33	2.89	2.93	2.84
query34	16.87	4.34	4.31
query35	4.45	4.42	4.44
query36	0.66	0.46	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.15	0.12	0.12
query41	0.09	0.06	0.05
query42	0.07	0.05	0.06
query43	0.04	0.04	0.04
Total cold run time: 109.28 s
Total hot run time: 31.08 s

Copy link
Contributor

PR approved by anyone and no changes requested.

morningman
morningman previously approved these changes Jul 18, 2024
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39860 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a114ba093c180b165b8b70a48e52323cd72d89c1, data reload: false

------ Round 1 ----------------------------------
q1	17617	4318	4253	4253
q2	2011	191	184	184
q3	10441	1213	1043	1043
q4	10183	766	865	766
q5	7532	2697	2674	2674
q6	220	138	140	138
q7	955	626	628	626
q8	9230	2066	2051	2051
q9	8780	6565	6568	6565
q10	8887	3778	3776	3776
q11	468	244	240	240
q12	410	231	229	229
q13	18911	2973	3003	2973
q14	286	235	226	226
q15	535	496	494	494
q16	509	394	382	382
q17	983	670	659	659
q18	8202	7428	7370	7370
q19	5618	1383	1294	1294
q20	666	326	332	326
q21	4956	3303	3845	3303
q22	351	288	294	288
Total cold run time: 117751 ms
Total hot run time: 39860 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4465	4268	4270	4268
q2	370	279	268	268
q3	3054	2936	2955	2936
q4	2012	1702	1758	1702
q5	5652	5577	5452	5452
q6	236	133	135	133
q7	2304	1864	1835	1835
q8	3257	3436	3461	3436
q9	8799	8837	8769	8769
q10	4173	3753	3924	3753
q11	616	505	493	493
q12	840	643	630	630
q13	17341	3194	3188	3188
q14	312	309	285	285
q15	533	485	500	485
q16	494	433	445	433
q17	1807	1490	1502	1490
q18	8250	8008	7986	7986
q19	3414	1645	1723	1645
q20	2342	1885	1852	1852
q21	5145	4984	5068	4984
q22	573	520	524	520
Total cold run time: 75989 ms
Total hot run time: 56543 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173757 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a114ba093c180b165b8b70a48e52323cd72d89c1, data reload: false

query1	914	371	360	360
query2	6427	1900	1870	1870
query3	6651	203	212	203
query4	28181	17181	17259	17181
query5	3644	493	477	477
query6	265	171	155	155
query7	4579	287	288	287
query8	241	189	188	188
query9	8476	2542	2497	2497
query10	443	272	276	272
query11	10600	10176	9878	9878
query12	108	79	83	79
query13	1664	361	387	361
query14	10191	7638	7697	7638
query15	212	158	165	158
query16	7706	332	323	323
query17	1700	523	527	523
query18	1920	270	298	270
query19	206	153	157	153
query20	88	86	82	82
query21	213	126	128	126
query22	4268	4012	3923	3923
query23	34128	33584	33803	33584
query24	10877	2902	2989	2902
query25	635	412	410	410
query26	715	167	157	157
query27	2254	288	288	288
query28	5932	2127	2101	2101
query29	906	666	672	666
query30	256	153	172	153
query31	994	757	772	757
query32	94	59	57	57
query33	699	315	328	315
query34	901	494	512	494
query35	705	602	579	579
query36	1118	970	987	970
query37	157	95	90	90
query38	3060	2850	2854	2850
query39	903	857	853	853
query40	201	128	125	125
query41	50	51	48	48
query42	115	108	102	102
query43	531	481	488	481
query44	1169	743	734	734
query45	193	163	166	163
query46	1083	745	725	725
query47	1846	1754	1765	1754
query48	382	308	303	303
query49	849	428	437	428
query50	781	410	401	401
query51	6862	6738	6788	6738
query52	107	104	105	104
query53	377	304	300	300
query54	892	464	460	460
query55	77	77	75	75
query56	319	283	286	283
query57	1150	1085	1058	1058
query58	274	271	291	271
query59	2957	2756	2706	2706
query60	330	301	300	300
query61	121	113	112	112
query62	802	653	638	638
query63	318	293	304	293
query64	9312	2187	1688	1688
query65	3244	3097	3092	3092
query66	791	319	324	319
query67	15512	14847	14881	14847
query68	8445	563	563	563
query69	730	505	379	379
query70	1318	1160	1151	1151
query71	527	285	285	285
query72	9404	5920	5668	5668
query73	2267	329	332	329
query74	6023	5613	5686	5613
query75	5177	2682	2696	2682
query76	5205	1058	900	900
query77	791	312	310	310
query78	10449	9026	8858	8858
query79	11021	538	528	528
query80	1280	483	477	477
query81	583	224	214	214
query82	524	143	132	132
query83	323	165	168	165
query84	277	136	87	87
query85	918	311	297	297
query86	365	299	304	299
query87	3351	3108	3139	3108
query88	4769	2467	2444	2444
query89	530	372	384	372
query90	2056	191	191	191
query91	126	99	101	99
query92	63	48	48	48
query93	6706	517	512	512
query94	1387	226	226	226
query95	403	314	328	314
query96	614	269	274	269
query97	3208	3040	2991	2991
query98	216	238	190	190
query99	1551	1278	1289	1278
Total cold run time: 304283 ms
Total hot run time: 173757 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a114ba093c180b165b8b70a48e52323cd72d89c1, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.09	0.09
query5	0.50	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.55	0.49	0.48
query10	0.55	0.55	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.77	0.77
query15	0.86	0.82	0.80
query16	0.36	0.36	0.35
query17	0.99	0.97	0.97
query18	0.24	0.23	0.22
query19	1.78	1.67	1.74
query20	0.02	0.01	0.01
query21	15.42	0.76	0.65
query22	3.72	6.98	2.22
query23	18.28	1.35	1.20
query24	2.23	0.23	0.21
query25	0.17	0.09	0.09
query26	0.29	0.21	0.22
query27	0.46	0.25	0.25
query28	13.18	1.01	1.00
query29	12.62	3.32	3.37
query30	0.25	0.06	0.06
query31	2.87	0.39	0.38
query32	3.28	0.49	0.48
query33	2.92	2.92	2.87
query34	17.21	4.35	4.39
query35	4.44	4.39	4.41
query36	0.65	0.46	0.51
query37	0.20	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.15	0.12	0.13
query41	0.09	0.06	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.44 s
Total hot run time: 30.84 s

Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 18, 2024
@morningman morningman merged commit 5af1195 into apache:master Jul 18, 2024
26 of 27 checks passed
morningman added a commit to morningman/doris that referenced this pull request Jul 18, 2024
…e hive write transaction statistics and (apache#37191)

iceberg support insert overwrite
1. Iceberg transaction adds insert overwrite feature
2.Iceberg transaction merge the commit functions of table and partition
3.Use class SimpleTableInfo to replace the internal class
DatabaseTableName of HmsTransaction
4.Unified the file statistics of hive and iceberg write

---------

Co-authored-by: lik40 <lik40@chinatelecom.cn>
Co-authored-by: morningman <morningman@163.com>
morningman added a commit that referenced this pull request Jul 19, 2024
…e hive write transaction statistics and (#37191) (#38097)

bp #37191

Co-authored-by: kang <35803862+ghkang98@users.noreply.github.com>
Co-authored-by: lik40 <lik40@chinatelecom.cn>
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
…e hive write transaction statistics and (#37191)

iceberg support insert overwrite
1. Iceberg transaction adds insert overwrite feature
2.Iceberg transaction merge the commit functions of table and partition
3.Use class SimpleTableInfo to replace the internal class
DatabaseTableName of HmsTransaction
4.Unified the file statistics of hive and iceberg write

---------

Co-authored-by: lik40 <lik40@chinatelecom.cn>
Co-authored-by: morningman <morningman@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants