Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](inverted index) opt value extraction from column to string (#37395) #38937

Merged
merged 1 commit into from
Aug 13, 2024
Merged

[opt](inverted index) opt value extraction from column to string (#37395) #38937

merged 1 commit into from
Aug 13, 2024

Conversation

zzzxl1993
Copy link
Contributor

@zzzxl1993
[opt](inverted index) opt value extraction from column to string (apa…
e3714eb 
…che#37395)

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zzzxl1993
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 6, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49353 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e3714ebce528f631682203e59482e0464c3f40fe, data reload: false

------ Round 1 ----------------------------------
q1	17812	4442	4309	4309
q2	2059	161	147	147
q3	10270	1878	1929	1878
q4	10345	1244	1333	1244
q5	8478	3780	3929	3780
q6	239	126	149	126
q7	2054	1621	1595	1595
q8	9412	2725	2691	2691
q9	10398	10153	10170	10153
q10	8634	3486	3525	3486
q11	418	238	240	238
q12	465	302	299	299
q13	18337	3969	4003	3969
q14	355	334	347	334
q15	503	453	452	452
q16	670	562	571	562
q17	1136	955	966	955
q18	7246	6938	6902	6902
q19	1675	1597	1510	1510
q20	517	314	317	314
q21	4425	4081	4022	4022
q22	487	387	412	387
Total cold run time: 115935 ms
Total hot run time: 49353 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4334	4295	4280	4280
q2	320	222	228	222
q3	4179	4177	4125	4125
q4	2765	2738	2745	2738
q5	7148	7118	7117	7117
q6	235	121	118	118
q7	3248	2798	2851	2798
q8	4332	4464	4475	4464
q9	16845	16676	16790	16676
q10	4234	4280	4257	4257
q11	769	708	735	708
q12	1030	839	863	839
q13	6953	3748	3763	3748
q14	448	423	416	416
q15	505	455	452	452
q16	738	697	678	678
q17	3886	3837	3881	3837
q18	8842	8675	8814	8675
q19	1710	1714	1646	1646
q20	2354	2128	2091	2091
q21	8492	8470	8457	8457
q22	985	932	897	897
Total cold run time: 84352 ms
Total hot run time: 79239 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.75% (8104/21466)
Line Coverage: 29.39% (66377/225861)
Region Coverage: 28.90% (34256/118539)
Branch Coverage: 24.78% (17604/71034)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e3714ebce528f631682203e59482e0464c3f40fe_e3714ebce528f631682203e59482e0464c3f40fe/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202382 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e3714ebce528f631682203e59482e0464c3f40fe, data reload: false

query1	931	387	408	387
query2	6549	2263	2162	2162
query3	7063	212	203	203
query4	20254	18242	18373	18242
query5	22525	6532	6527	6527
query6	291	220	243	220
query7	4153	310	314	310
query8	252	258	240	240
query9	3143	2682	2627	2627
query10	404	301	280	280
query11	11506	10937	11613	10937
query12	143	80	89	80
query13	5940	668	674	668
query14	19774	13623	13786	13623
query15	383	222	222	222
query16	6904	277	274	274
query17	1895	1467	882	882
query18	2345	409	407	407
query19	212	144	157	144
query20	80	77	79	77
query21	191	100	89	89
query22	5393	4880	5163	4880
query23	32370	31660	32045	31660
query24	6802	6563	6530	6530
query25	536	424	429	424
query26	521	164	160	160
query27	1795	300	302	300
query28	6077	2273	2230	2230
query29	2938	2734	2796	2734
query30	242	174	161	161
query31	919	725	734	725
query32	65	60	60	60
query33	414	260	267	260
query34	853	469	487	469
query35	1112	919	919	919
query36	1285	1114	1362	1114
query37	89	64	62	62
query38	3055	2971	2951	2951
query39	1379	1342	1331	1331
query40	205	99	100	99
query41	47	37	36	36
query42	89	80	100	80
query43	679	640	596	596
query44	1140	715	720	715
query45	240	226	226	226
query46	1223	967	956	956
query47	1774	1769	1601	1601
query48	982	699	680	680
query49	616	383	393	383
query50	860	629	585	585
query51	4792	4674	4709	4674
query52	88	84	76	76
query53	450	317	321	317
query54	2655	2466	2468	2466
query55	89	74	80	74
query56	221	233	208	208
query57	1133	1134	1196	1134
query58	204	205	197	197
query59	3686	3387	3334	3334
query60	215	206	209	206
query61	98	96	96	96
query62	881	484	473	473
query63	485	350	345	345
query64	2288	1517	1438	1438
query65	3664	3571	3563	3563
query66	769	398	381	381
query67	15407	15966	14955	14955
query68	8976	653	638	638
query69	565	375	355	355
query70	1831	1564	1451	1451
query71	428	314	304	304
query72	6753	3511	3479	3479
query73	747	326	318	318
query74	6277	5906	5835	5835
query75	5351	3642	3671	3642
query76	5283	1148	1141	1141
query77	884	257	262	257
query78	12588	12178	11872	11872
query79	12314	641	636	636
query80	853	399	401	399
query81	498	235	237	235
query82	1020	97	97	97
query83	186	130	131	130
query84	269	73	69	69
query85	892	322	314	314
query86	339	327	292	292
query87	3227	3024	2995	2995
query88	5705	2305	2305	2305
query89	394	297	286	286
query90	2534	217	206	206
query91	160	123	121	121
query92	58	54	54	54
query93	5970	556	591	556
query94	1295	213	208	208
query95	1123	1079	1072	1072
query96	629	336	324	324
query97	6365	6327	6365	6327
query98	189	171	175	171
query99	3091	828	939	828
Total cold run time: 322557 ms
Total hot run time: 202382 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.25 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e3714ebce528f631682203e59482e0464c3f40fe, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.03	0.02
query3	0.24	0.05	0.05
query4	1.79	0.07	0.06
query5	0.53	0.53	0.52
query6	1.23	0.61	0.62
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.53	0.48	0.47
query10	0.53	0.54	0.53
query11	0.12	0.09	0.09
query12	0.12	0.09	0.09
query13	0.63	0.62	0.60
query14	0.77	0.78	0.78
query15	0.77	0.77	0.75
query16	0.38	0.35	0.37
query17	1.01	1.05	1.02
query18	0.23	0.25	0.25
query19	1.88	1.85	1.83
query20	0.02	0.01	0.02
query21	15.47	0.56	0.56
query22	1.74	2.27	2.12
query23	17.43	1.13	0.84
query24	6.99	1.07	1.40
query25	0.40	0.10	0.06
query26	0.72	0.17	0.16
query27	0.04	0.04	0.04
query28	5.90	0.72	0.75
query29	12.76	2.28	2.27
query30	0.62	0.56	0.54
query31	2.82	0.39	0.37
query32	3.41	0.50	0.49
query33	3.10	3.06	3.06
query34	15.26	4.83	4.78
query35	4.85	4.86	4.83
query36	1.02	1.01	1.01
query37	0.06	0.05	0.05
query38	0.04	0.03	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 103.83 s
Total hot run time: 31.25 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e3714ebce528f631682203e59482e0464c3f40fe with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@xiaokang xiaokang merged commit cf868f9 into apache:branch-2.0 Aug 13, 2024
22 of 25 checks passed
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@zzzxl1993
[opt](inverted index) opt value extraction from column to string (apa…
0b5ee9f 
…che#37395) (apache#38937)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@zzzxl1993 @doris-robot @xiaokang