Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dep](thirdparty) import wasmtime library #30591

Merged
merged 6 commits into from
Feb 1, 2024

Conversation

taptao
Copy link
Contributor

@taptao taptao commented Jan 30, 2024

Proposed changes

Issue Number: close #24504

Add wasm-lib thirdparty.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@taptao
Copy link
Contributor Author

taptao commented Jan 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37158 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f02685cd1375cdfd4fc4e8bcace47bf9262d1e50, data reload: false

------ Round 1 ----------------------------------
q1	17597	4500	4454	4454
q2	2057	140	135	135
q3	10596	935	922	922
q4	4983	752	754	752
q5	7833	2866	2893	2866
q6	189	123	125	123
q7	1138	726	734	726
q8	9839	2079	2046	2046
q9	8071	6449	6332	6332
q10	8098	2428	2442	2428
q11	417	211	198	198
q12	738	289	282	282
q13	17997	3315	3289	3289
q14	280	256	237	237
q15	528	493	476	476
q16	467	408	407	407
q17	939	543	603	543
q18	6868	6147	5956	5956
q19	1575	1390	1356	1356
q20	579	332	326	326
q21	6674	2995	3044	2995
q22	805	309	311	309
Total cold run time: 108268 ms
Total hot run time: 37158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4532	4395	4549	4395
q2	326	228	233	228
q3	3023	2905	2858	2858
q4	1899	1654	1706	1654
q5	5153	5204	5207	5204
q6	194	117	115	115
q7	2165	1768	1747	1747
q8	3098	3230	3264	3230
q9	8353	8218	8253	8218
q10	5800	3587	3504	3504
q11	530	460	446	446
q12	749	600	591	591
q13	9350	3123	3094	3094
q14	278	243	266	243
q15	533	492	491	491
q16	507	484	479	479
q17	1828	1626	1687	1626
q18	7912	7733	7620	7620
q19	6870	1518	1516	1516
q20	2135	1904	1914	1904
q21	4756	4581	4449	4449
q22	538	475	490	475
Total cold run time: 70529 ms
Total hot run time: 54087 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8626/23835)
Line Coverage: 28.25% (70544/249685)
Region Coverage: 27.25% (36404/133580)
Branch Coverage: 24.03% (18642/77562)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f02685cd1375cdfd4fc4e8bcace47bf9262d1e50_f02685cd1375cdfd4fc4e8bcace47bf9262d1e50/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172791 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f02685cd1375cdfd4fc4e8bcace47bf9262d1e50, data reload: false

query1	923	334	325	325
query2	6568	2060	2099	2060
query3	6704	211	207	207
query4	27295	22039	21956	21956
query5	4455	417	359	359
query6	249	163	160	160
query7	4603	257	263	257
query8	240	177	178	177
query9	9154	2251	2233	2233
query10	404	198	204	198
query11	16418	15408	15358	15358
query12	118	68	60	60
query13	1683	371	363	363
query14	9243	6564	6534	6534
query15	203	173	179	173
query16	6171	259	249	249
query17	1817	463	472	463
query18	1780	248	245	245
query19	173	131	129	129
query20	69	73	68	68
query21	199	139	135	135
query22	4872	4796	4746	4746
query23	31294	30234	30292	30234
query24	12499	2709	2765	2709
query25	604	309	311	309
query26	2082	130	136	130
query27	3203	286	285	285
query28	7848	1814	1801	1801
query29	2314	629	617	617
query30	289	134	139	134
query31	915	693	717	693
query32	86	53	49	49
query33	701	211	210	210
query34	1198	443	460	443
query35	870	773	756	756
query36	1261	1237	1139	1139
query37	173	55	60	55
query38	3250	3155	3102	3102
query39	1321	1252	1242	1242
query40	333	90	83	83
query41	38	37	34	34
query42	88	81	87	81
query43	538	463	509	463
query44	1098	680	677	677
query45	190	181	180	180
query46	1049	631	629	629
query47	1542	1514	1547	1514
query48	384	313	324	313
query49	1278	278	275	275
query50	675	306	319	306
query51	5304	5141	5127	5127
query52	90	84	76	76
query53	322	269	271	269
query54	248	176	184	176
query55	75	80	74	74
query56	173	168	162	162
query57	1040	923	919	919
query58	187	151	155	151
query59	2454	2296	2268	2268
query60	194	175	169	169
query61	80	81	80	80
query62	652	372	365	365
query63	277	263	254	254
query64	5653	1723	1709	1709
query65	3266	3203	3201	3201
query66	1494	321	314	314
query67	14291	14139	14188	14139
query68	3814	500	492	492
query69	416	305	309	305
query70	1530	1536	1584	1536
query71	291	220	208	208
query72	3638	2836	2796	2796
query73	676	319	321	319
query74	6626	6166	6250	6166
query75	2822	2321	2325	2321
query76	2867	994	969	969
query77	346	235	232	232
query78	9043	8678	8560	8560
query79	2675	512	493	493
query80	2024	315	306	306
query81	525	198	203	198
query82	778	78	83	78
query83	244	136	123	123
query84	277	70	74	70
query85	2357	320	316	316
query86	504	409	385	385
query87	3490	3228	3232	3228
query88	3989	2265	2272	2265
query89	413	355	369	355
query90	1930	189	188	188
query91	151	116	116	116
query92	56	45	42	42
query93	2830	441	428	428
query94	1340	163	161	161
query95	641	470	470	470
query96	619	330	331	330
query97	4272	4131	4093	4093
query98	225	206	189	189
query99	1050	692	682	682
Total cold run time: 278721 ms
Total hot run time: 172791 ms

zhangstar333
zhangstar333 previously approved these changes Jan 30, 2024
@doris-robot
Copy link

ClickBench: Total hot run time: 30.15 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f02685cd1375cdfd4fc4e8bcace47bf9262d1e50, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.51
query6	1.19	0.67	0.64
query7	0.02	0.01	0.02
query8	0.03	0.03	0.02
query9	0.56	0.52	0.48
query10	0.56	0.54	0.55
query11	0.12	0.08	0.08
query12	0.11	0.08	0.09
query13	0.61	0.60	0.60
query14	0.78	0.80	0.80
query15	0.79	0.78	0.78
query16	0.38	0.38	0.38
query17	1.02	1.01	1.02
query18	0.22	0.26	0.24
query19	1.82	1.78	1.75
query20	0.01	0.01	0.01
query21	15.41	0.60	0.57
query22	2.52	2.76	1.36
query23	17.30	0.73	0.79
query24	2.48	1.41	0.83
query25	0.45	0.09	0.06
query26	0.56	0.13	0.13
query27	0.06	0.06	0.05
query28	11.88	0.87	0.84
query29	12.50	3.06	3.04
query30	0.66	0.59	0.52
query31	2.78	0.34	0.37
query32	3.36	0.48	0.48
query33	3.22	3.24	3.22
query34	16.00	4.38	4.25
query35	4.36	4.32	4.27
query36	1.08	1.07	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.14	0.13
query41	0.07	0.01	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 105.74 s
Total hot run time: 30.15 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 30, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f02685cd1375cdfd4fc4e8bcace47bf9262d1e50 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.3 seconds inserted 10000000 Rows, about 751K ops/s

@taptao
Copy link
Contributor Author

taptao commented Jan 30, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8624/23831)
Line Coverage: 28.25% (70513/249610)
Region Coverage: 27.24% (36379/133546)
Branch Coverage: 24.03% (18632/77538)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae82f16a9555befd39f792ba9a84049cb0c26612_ae82f16a9555befd39f792ba9a84049cb0c26612/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36586 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

------ Round 1 ----------------------------------
q1	17679	4580	4429	4429
q2	2041	143	130	130
q3	10673	904	948	904
q4	4650	714	716	714
q5	7699	2816	2751	2751
q6	184	119	120	119
q7	1114	714	709	709
q8	9267	1963	1997	1963
q9	7155	6325	6255	6255
q10	8040	2430	2395	2395
q11	412	206	189	189
q12	767	292	277	277
q13	18022	3287	3255	3255
q14	274	239	251	239
q15	526	497	480	480
q16	473	413	412	412
q17	923	512	506	506
q18	6702	5940	5891	5891
q19	1532	1357	1352	1352
q20	581	348	326	326
q21	7142	3022	2995	2995
q22	805	295	302	295
Total cold run time: 106661 ms
Total hot run time: 36586 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4536	4420	4357	4357
q2	332	217	228	217
q3	2982	2849	2816	2816
q4	1874	1680	1718	1680
q5	5183	5262	5236	5236
q6	190	112	114	112
q7	2192	1798	1766	1766
q8	3043	3200	3227	3200
q9	8322	8293	8311	8293
q10	5819	3563	3617	3563
q11	538	446	465	446
q12	757	591	607	591
q13	11409	3103	3084	3084
q14	265	247	254	247
q15	532	493	490	490
q16	507	477	469	469
q17	1807	1690	1671	1671
q18	7957	7777	7381	7381
q19	6331	1477	1554	1477
q20	2118	1915	1894	1894
q21	4729	4682	4624	4624
q22	566	440	493	440
Total cold run time: 71989 ms
Total hot run time: 54054 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174150 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

query1	936	330	328	328
query2	6581	2025	1851	1851
query3	6713	212	199	199
query4	26515	22212	22184	22184
query5	4457	418	418	418
query6	253	157	154	154
query7	4618	261	265	261
query8	252	178	175	175
query9	8929	2265	2266	2265
query10	424	205	212	205
query11	16981	15480	15540	15480
query12	120	72	61	61
query13	1672	376	375	375
query14	9570	6635	6539	6539
query15	238	185	176	176
query16	6551	260	254	254
query17	1886	474	471	471
query18	1936	268	271	268
query19	197	143	155	143
query20	86	89	79	79
query21	205	139	137	137
query22	5463	5314	5437	5314
query23	31974	30443	30249	30249
query24	11098	2775	2790	2775
query25	544	312	305	305
query26	824	136	137	136
query27	2798	289	288	288
query28	6374	1829	1822	1822
query29	941	623	603	603
query30	280	138	141	138
query31	907	709	715	709
query32	91	51	50	50
query33	681	210	212	210
query34	1006	451	467	451
query35	844	750	785	750
query36	1300	1231	1212	1212
query37	90	59	62	59
query38	3303	3134	3149	3134
query39	1303	1250	1231	1231
query40	192	87	87	87
query41	38	36	34	34
query42	91	82	84	82
query43	519	473	495	473
query44	1071	675	684	675
query45	198	177	171	171
query46	1032	643	663	643
query47	1528	1500	1522	1500
query48	373	303	303	303
query49	1116	294	273	273
query50	689	310	309	309
query51	5269	5173	5183	5173
query52	100	82	88	82
query53	323	267	271	267
query54	256	190	198	190
query55	83	78	78	78
query56	179	166	169	166
query57	924	862	913	862
query58	191	164	164	164
query59	2368	2233	2337	2233
query60	201	181	186	181
query61	88	86	84	84
query62	658	348	354	348
query63	276	259	256	256
query64	4216	1778	1749	1749
query65	3315	3208	3190	3190
query66	1145	344	314	314
query67	14453	14226	14121	14121
query68	5399	518	503	503
query69	454	326	313	313
query70	1560	1542	1557	1542
query71	280	212	212	212
query72	4111	2876	2866	2866
query73	699	319	314	314
query74	6638	6314	6314	6314
query75	2858	2279	2325	2279
query76	3711	954	1001	954
query77	360	239	232	232
query78	9201	8633	8560	8560
query79	2168	502	498	498
query80	1419	325	321	321
query81	539	197	200	197
query82	671	82	80	80
query83	246	121	119	119
query84	236	74	78	74
query85	1929	351	337	337
query86	510	421	380	380
query87	3415	3258	3262	3258
query88	3763	2290	2279	2279
query89	426	359	339	339
query90	1882	187	188	187
query91	149	121	120	120
query92	51	48	46	46
query93	1973	432	441	432
query94	1334	157	159	157
query95	507	459	459	459
query96	621	327	324	324
query97	4231	4160	4157	4157
query98	204	190	183	183
query99	970	647	675	647
Total cold run time: 272348 ms
Total hot run time: 174150 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.53 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

query1	0.04	0.03	0.03
query2	0.05	0.02	0.02
query3	0.23	0.06	0.06
query4	1.67	0.10	0.10
query5	0.52	0.51	0.51
query6	1.20	0.65	0.66
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.54	0.50	0.50
query10	0.55	0.55	0.56
query11	0.12	0.09	0.08
query12	0.12	0.09	0.10
query13	0.60	0.61	0.61
query14	0.80	0.80	0.81
query15	0.82	0.78	0.77
query16	0.37	0.40	0.39
query17	1.02	1.00	0.99
query18	0.21	0.28	0.25
query19	1.80	1.81	1.80
query20	0.01	0.00	0.01
query21	15.40	0.54	0.56
query22	2.53	2.29	1.78
query23	17.29	0.84	0.84
query24	2.91	1.36	1.42
query25	0.39	0.20	0.19
query26	0.52	0.15	0.14
query27	0.06	0.05	0.04
query28	10.94	0.89	0.85
query29	12.52	3.09	3.20
query30	0.62	0.55	0.58
query31	2.79	0.34	0.35
query32	3.36	0.49	0.48
query33	3.19	3.19	3.21
query34	15.72	4.36	4.33
query35	4.38	4.31	4.29
query36	1.10	1.04	1.07
query37	0.06	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.01
query40	0.16	0.13	0.12
query41	0.06	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 104.84 s
Total hot run time: 31.53 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

@taptao
Copy link
Contributor Author

taptao commented Jan 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36710 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

------ Round 1 ----------------------------------
q1	17702	4415	4406	4406
q2	2037	139	132	132
q3	10606	917	928	917
q4	4649	766	714	714
q5	7672	2775	2815	2775
q6	187	120	118	118
q7	1121	719	707	707
q8	9298	2007	2014	2007
q9	7268	6355	6335	6335
q10	8103	2422	2447	2422
q11	410	208	197	197
q12	759	266	277	266
q13	18013	3330	3284	3284
q14	281	247	246	246
q15	524	496	481	481
q16	481	398	417	398
q17	979	529	459	459
q18	6916	5933	5897	5897
q19	1572	1401	1309	1309
q20	589	341	325	325
q21	6855	3031	3113	3031
q22	815	297	284	284
Total cold run time: 106837 ms
Total hot run time: 36710 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4417	4441	4415	4415
q2	342	234	234	234
q3	2975	2830	2832	2830
q4	1822	1643	1695	1643
q5	5126	5286	5289	5286
q6	189	113	114	113
q7	2182	1794	1786	1786
q8	3100	3232	3251	3232
q9	8336	8371	8332	8332
q10	5875	3634	3571	3571
q11	548	464	464	464
q12	750	593	602	593
q13	14139	3043	3086	3043
q14	279	251	264	251
q15	542	484	493	484
q16	506	496	484	484
q17	1825	1668	1702	1668
q18	8185	7675	7650	7650
q19	9144	1534	1550	1534
q20	2164	1908	1899	1899
q21	5567	4494	4555	4494
q22	535	426	466	426
Total cold run time: 78548 ms
Total hot run time: 54432 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.19% (8624/23831)
Line Coverage: 28.25% (70530/249635)
Region Coverage: 27.25% (36391/133566)
Branch Coverage: 24.03% (18639/77556)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae82f16a9555befd39f792ba9a84049cb0c26612_ae82f16a9555befd39f792ba9a84049cb0c26612/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 172575 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

query1	933	337	332	332
query2	6566	2075	1895	1895
query3	6702	203	197	197
query4	27634	22021	22122	22021
query5	4459	423	359	359
query6	257	164	164	164
query7	4617	265	253	253
query8	246	170	175	170
query9	8977	2261	2257	2257
query10	407	189	206	189
query11	16828	15297	15327	15297
query12	126	68	62	62
query13	1675	364	367	364
query14	9242	6518	6432	6432
query15	210	177	186	177
query16	6373	264	249	249
query17	1874	482	464	464
query18	1901	254	252	252
query19	181	137	130	130
query20	73	67	69	67
query21	205	131	128	128
query22	4981	4753	4392	4392
query23	31183	30350	30273	30273
query24	11068	2718	2744	2718
query25	544	312	306	306
query26	841	138	139	138
query27	2836	290	282	282
query28	6207	1842	1835	1835
query29	947	618	627	618
query30	281	136	139	136
query31	910	699	709	699
query32	89	51	47	47
query33	592	208	211	208
query34	827	458	457	457
query35	881	776	765	765
query36	1329	1160	1172	1160
query37	92	58	60	58
query38	3271	3155	3152	3152
query39	1316	1236	1253	1236
query40	202	90	80	80
query41	39	33	35	33
query42	93	80	84	80
query43	563	490	498	490
query44	1033	675	686	675
query45	191	178	175	175
query46	1050	627	655	627
query47	1537	1512	1519	1512
query48	400	304	303	303
query49	1114	283	277	277
query50	693	304	307	304
query51	5224	5197	5199	5197
query52	98	81	79	79
query53	320	271	254	254
query54	264	187	185	185
query55	79	73	76	73
query56	181	162	172	162
query57	974	916	846	846
query58	180	155	154	154
query59	2480	2472	2473	2472
query60	197	180	179	179
query61	86	85	92	85
query62	621	361	355	355
query63	275	259	257	257
query64	4163	1763	1718	1718
query65	3292	3232	3193	3193
query66	1130	318	311	311
query67	14496	14222	14248	14222
query68	3962	509	498	498
query69	427	313	302	302
query70	1548	1484	1494	1484
query71	278	217	216	216
query72	3703	2830	2876	2830
query73	686	311	323	311
query74	6603	6257	6239	6239
query75	2803	2276	2282	2276
query76	2892	1031	906	906
query77	358	231	237	231
query78	9035	8683	8558	8558
query79	2226	489	492	489
query80	1463	324	330	324
query81	539	195	194	194
query82	671	85	75	75
query83	236	121	112	112
query84	234	75	76	75
query85	1832	342	344	342
query86	521	399	388	388
query87	3427	3274	3262	3262
query88	3917	2182	2181	2181
query89	420	390	343	343
query90	1959	186	185	185
query91	149	121	119	119
query92	53	45	44	44
query93	2057	416	407	407
query94	1364	161	161	161
query95	515	455	455	455
query96	617	324	312	312
query97	4257	4124	4145	4124
query98	210	193	181	181
query99	1018	753	714	714
Total cold run time: 268636 ms
Total hot run time: 172575 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.07 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.03
query3	0.23	0.07	0.06
query4	1.66	0.10	0.09
query5	0.52	0.51	0.52
query6	1.23	0.66	0.64
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.55	0.50	0.52
query10	0.55	0.55	0.53
query11	0.12	0.09	0.08
query12	0.11	0.09	0.10
query13	0.60	0.59	0.60
query14	0.79	0.79	0.81
query15	0.79	0.78	0.77
query16	0.40	0.39	0.38
query17	1.01	1.00	0.99
query18	0.21	0.29	0.23
query19	1.88	1.80	1.84
query20	0.01	0.01	0.02
query21	15.43	0.56	0.56
query22	2.68	2.53	1.99
query23	17.31	0.82	0.83
query24	2.49	1.43	0.94
query25	0.35	0.22	0.09
query26	0.59	0.14	0.13
query27	0.04	0.07	0.03
query28	11.55	0.86	0.85
query29	12.49	3.17	3.12
query30	0.67	0.58	0.52
query31	2.79	0.35	0.34
query32	3.35	0.48	0.48
query33	3.19	3.20	3.19
query34	15.98	4.28	4.29
query35	4.25	4.28	4.31
query36	1.12	1.06	1.06
query37	0.07	0.05	0.05
query38	0.04	0.03	0.03
query39	0.03	0.02	0.01
query40	0.16	0.14	0.12
query41	0.07	0.01	0.02
query42	0.03	0.01	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.52 s
Total hot run time: 31.07 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ae82f16a9555befd39f792ba9a84049cb0c26612 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

dataroaring
dataroaring previously approved these changes Jan 31, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 1, 2024
@taptao
Copy link
Contributor Author

taptao commented Feb 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 36609 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a, data reload: false

------ Round 1 ----------------------------------
q1	17654	4458	4435	4435
q2	2045	148	127	127
q3	10599	909	905	905
q4	4653	678	687	678
q5	7676	2746	2784	2746
q6	183	117	118	117
q7	1128	711	701	701
q8	9315	1995	1995	1995
q9	7205	6293	6262	6262
q10	8107	2403	2413	2403
q11	419	211	194	194
q12	755	291	270	270
q13	18004	3291	3279	3279
q14	279	239	247	239
q15	528	491	495	491
q16	481	429	401	401
q17	911	547	540	540
q18	6884	6000	5881	5881
q19	1558	1346	1329	1329
q20	608	325	335	325
q21	7046	3030	3000	3000
q22	809	291	293	291
Total cold run time: 106847 ms
Total hot run time: 36609 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4341	4391	4341
q2	319	236	234	234
q3	2938	2841	2782	2782
q4	1834	1672	1654	1654
q5	5185	5191	5219	5191
q6	191	112	114	112
q7	2146	1742	1783	1742
q8	3046	3217	3234	3217
q9	8257	8346	8277	8277
q10	5978	3564	3569	3564
q11	548	454	472	454
q12	743	576	606	576
q13	12918	3093	3092	3092
q14	274	254	258	254
q15	543	499	491	491
q16	523	490	469	469
q17	1828	1681	1675	1675
q18	7972	7687	7599	7599
q19	10262	1556	1507	1507
q20	2152	1942	1920	1920
q21	5080	4602	4840	4602
q22	573	492	474	474
Total cold run time: 77706 ms
Total hot run time: 54227 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175415 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a, data reload: false

query1	2981	349	331	331
query2	7684	2021	1878	1878
query3	9959	222	200	200
query4	32807	22102	22069	22069
query5	4457	408	425	408
query6	245	158	154	154
query7	4599	270	263	263
query8	259	173	176	173
query9	9114	2255	2243	2243
query10	415	208	208	208
query11	18353	15344	15206	15206
query12	120	71	64	64
query13	1621	367	367	367
query14	9818	7121	7500	7121
query15	241	174	183	174
query16	7501	270	251	251
query17	1704	494	471	471
query18	1922	256	249	249
query19	333	130	129	129
query20	73	66	70	66
query21	204	128	131	128
query22	5106	4873	4941	4873
query23	31154	30339	30295	30295
query24	7762	2753	2742	2742
query25	510	312	296	296
query26	735	139	142	139
query27	2144	285	289	285
query28	5435	1843	1825	1825
query29	937	623	601	601
query30	279	131	133	131
query31	924	717	707	707
query32	92	52	51	51
query33	463	207	203	203
query34	837	452	468	452
query35	845	763	749	749
query36	1326	1258	1266	1258
query37	91	57	54	54
query38	3301	3161	3156	3156
query39	1310	1257	1233	1233
query40	195	84	80	80
query41	36	35	33	33
query42	86	81	81	81
query43	504	493	491	491
query44	1011	672	705	672
query45	196	180	173	173
query46	1044	620	625	620
query47	1623	1530	1550	1530
query48	399	314	311	311
query49	1117	291	279	279
query50	689	303	312	303
query51	5312	5087	5167	5087
query52	93	80	78	78
query53	317	264	249	249
query54	212	172	185	172
query55	80	74	80	74
query56	168	161	172	161
query57	950	905	916	905
query58	185	159	159	159
query59	2412	2394	2176	2176
query60	200	178	176	176
query61	85	86	90	86
query62	620	366	333	333
query63	277	260	260	260
query64	4719	3707	3447	3447
query65	3277	3259	3218	3218
query66	943	320	308	308
query67	14406	14437	14292	14292
query68	5448	500	500	500
query69	454	313	304	304
query70	1530	1604	1591	1591
query71	281	211	202	202
query72	6146	3116	2865	2865
query73	690	316	321	316
query74	6794	6330	6356	6330
query75	3012	2273	2299	2273
query76	2999	939	980	939
query77	385	223	226	223
query78	9237	8535	8603	8535
query79	3309	501	498	498
query80	2150	327	319	319
query81	538	194	194	194
query82	872	87	76	76
query83	388	118	119	118
query84	285	75	69	69
query85	1650	337	317	317
query86	512	403	384	384
query87	3498	3278	3296	3278
query88	4244	2145	2140	2140
query89	439	362	371	362
query90	1898	184	180	180
query91	150	118	115	115
query92	51	42	41	41
query93	4781	460	448	448
query94	1368	161	158	158
query95	517	468	460	460
query96	634	315	306	306
query97	4261	4078	4115	4078
query98	221	201	188	188
query99	1133	651	694	651
Total cold run time: 288692 ms
Total hot run time: 175415 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a, data reload: false

query1	0.04	0.03	0.03
query2	0.06	0.02	0.02
query3	0.22	0.06	0.06
query4	1.66	0.10	0.09
query5	0.53	0.52	0.51
query6	1.17	0.63	0.64
query7	0.01	0.01	0.01
query8	0.04	0.03	0.03
query9	0.56	0.50	0.51
query10	0.57	0.56	0.57
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.60	0.60	0.60
query14	0.79	0.81	0.78
query15	0.79	0.79	0.76
query16	0.39	0.39	0.39
query17	1.02	1.03	1.03
query18	0.21	0.26	0.23
query19	1.81	1.75	1.78
query20	0.01	0.00	0.00
query21	15.38	0.57	0.56
query22	2.58	1.97	2.20
query23	17.58	0.97	0.85
query24	2.64	1.29	0.46
query25	0.33	0.11	0.25
query26	0.46	0.14	0.15
query27	0.06	0.05	0.04
query28	11.91	0.86	0.84
query29	12.49	3.16	3.20
query30	0.64	0.55	0.54
query31	2.79	0.36	0.36
query32	3.34	0.49	0.48
query33	3.21	3.24	3.23
query34	15.69	4.26	4.27
query35	4.31	4.31	4.35
query36	1.09	1.06	1.05
query37	0.07	0.06	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.01
query40	0.16	0.12	0.13
query41	0.08	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.02	0.01
Total cold run time: 105.62 s
Total hot run time: 30.76 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.18% (8626/23844)
Line Coverage: 28.24% (70546/249781)
Region Coverage: 27.25% (36407/133615)
Branch Coverage: 24.03% (18641/77588)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a_ae682cf39fd33e1952d3f5e6d1d6794d0ccc2e7a/report/index.html

Copy link
Contributor

@amorynan amorynan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 1, 2024
Copy link
Contributor

github-actions bot commented Feb 1, 2024

PR approved by at least one committer and no changes requested.

@dataroaring dataroaring merged commit 2490d6b into apache:master Feb 1, 2024
26 of 28 checks passed
@hello-stephen
Copy link
Contributor

I encounter an issue when compiling the latest thirdparty
cargo: command not found
may I ask what is the typical way to install cargo conveniently?
or can you just write it in the script where you use the command cargo?

@zclllyybb
Copy link
Contributor

@taptao FYI, maybe we could rethink the version of these libs because it may not comptable for some machine with old glibc version. some dependency is not avaliable.

Comment on lines +598 to +599
'WASMTIME'
'WASMTIME_CPP'
Copy link
Contributor

@adonis0147 adonis0147 Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @taptao , it seems that these libraries don't support macOS (arm64). I think you would better remove these lines here and add them somewhere else after checking the platform. Currently, this script can not be ran successfully on macOS (arm64).

https://docs.wasmtime.dev/stability-platform-support.html

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed this, but my env is mac arm, and the compilation was successful after installing rust, did you get any errors?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I noticed this, but my env is mac arm, and the compilation was successful after installing rust, did you get any errors?

Hi @taptao , thanks for your prompt reply. I ran the script successfully after setting Rust up.

@taptao
Copy link
Contributor Author

taptao commented Feb 2, 2024

I encounter an issue when compiling the latest thirdparty cargo: command not found may I ask what is the typical way to install cargo conveniently? or can you just write it in the script where you use the command cargo?

I installed rust through "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh"

@taptao
Copy link
Contributor Author

taptao commented Feb 2, 2024

@taptao FYI, maybe we could rethink the version of these libs because it may not comptable for some machine with old glibc version. some dependency is not avaliable.

yes, this is a big problem, i will rethink about it. any good suggestions?

HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Feb 4, 2024
HappenLee added a commit that referenced this pull request Feb 4, 2024
seawinde pushed a commit to seawinde/doris that referenced this pull request Feb 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants