Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature](json)support explode_json_object func #36887

Merged
merged 11 commits into from
Jul 2, 2024

Conversation

amorynan
Copy link
Contributor

Proposed changes

after this pr we support
expand json object into two column with key column (string) ,and value column(json value)

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

StringRef text = _json_object_column->get_data_at(row_idx);
if (text.data != nullptr) {
JsonbDocument* doc = JsonbDocument::createDocument(text.data, text.size);
if (UNLIKELY(!doc || !doc->getValue())) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: boolean expression can be simplified by DeMorgan's theorem [readability-simplify-boolean-expr]

        if (UNLIKELY(!doc || !doc->getValue())) {
            ^
Additional context

be/src/common/compiler_util.h:35: expanded from macro 'UNLIKELY'

#define UNLIKELY(expr) __builtin_expect(!!(expr), 0)
                                         ^

auto writer = std::make_unique<JsonbWriter>();
if (value->isObject()) {
_cur_size = value->length();
ObjectVal* obj = (ObjectVal*)value;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: use auto when initializing with a cast to avoid duplicating the type name [modernize-use-auto]

Suggested change
ObjectVal* obj = (ObjectVal*)value;
auto* obj = (ObjectVal*)value;


#pragma once

#include <stddef.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: inclusion of deprecated C++ header 'stddef.h'; consider using 'cstddef' instead [modernize-deprecated-headers]

Suggested change
#include <stddef.h>
#include <cstddef>

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40081 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99aaa3938997c35d8d9a8094960b50cd737e762b, data reload: false

------ Round 1 ----------------------------------
q1	17598	4487	4282	4282
q2	2032	191	192	191
q3	10464	1217	1134	1134
q4	10190	781	761	761
q5	7512	2649	2630	2630
q6	219	130	133	130
q7	957	602	600	600
q8	9222	2092	2094	2092
q9	9111	6524	6527	6524
q10	9010	3780	3781	3780
q11	458	234	235	234
q12	512	226	225	225
q13	17775	2987	2978	2978
q14	262	222	219	219
q15	528	473	471	471
q16	506	374	381	374
q17	988	714	685	685
q18	8045	7465	7455	7455
q19	2986	1493	1541	1493
q20	678	315	322	315
q21	4995	3172	3907	3172
q22	423	349	336	336
Total cold run time: 114471 ms
Total hot run time: 40081 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4234	4254	4234
q2	365	265	280	265
q3	2993	2766	2901	2766
q4	1954	1731	1706	1706
q5	5548	5585	5506	5506
q6	229	133	127	127
q7	2201	1806	1844	1806
q8	3277	3423	3430	3423
q9	8731	8664	8845	8664
q10	4050	3948	3787	3787
q11	591	492	503	492
q12	818	654	635	635
q13	16187	3145	3172	3145
q14	309	284	279	279
q15	540	494	488	488
q16	497	425	432	425
q17	1796	1539	1509	1509
q18	8092	7959	7869	7869
q19	1836	1609	1739	1609
q20	2701	1869	1895	1869
q21	5158	5051	4879	4879
q22	621	584	570	570
Total cold run time: 72857 ms
Total hot run time: 56053 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175070 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99aaa3938997c35d8d9a8094960b50cd737e762b, data reload: false

query1	924	393	379	379
query2	6462	2459	2405	2405
query3	6634	204	212	204
query4	18827	17564	17389	17389
query5	3599	461	487	461
query6	256	162	160	160
query7	4591	296	294	294
query8	341	297	305	297
query9	8571	2438	2415	2415
query10	547	282	271	271
query11	10590	10135	10154	10135
query12	113	89	81	81
query13	1634	365	361	361
query14	9948	6999	7889	6999
query15	230	185	184	184
query16	7586	266	268	266
query17	1902	557	517	517
query18	1501	282	270	270
query19	196	146	148	146
query20	89	83	81	81
query21	212	129	130	129
query22	4624	4347	4308	4308
query23	33788	33670	33671	33670
query24	11097	3012	2831	2831
query25	602	389	366	366
query26	724	158	157	157
query27	2318	326	328	326
query28	6281	2176	2184	2176
query29	866	634	665	634
query30	257	152	153	152
query31	985	756	752	752
query32	92	56	56	56
query33	751	298	291	291
query34	1025	470	491	470
query35	746	661	619	619
query36	1136	1001	991	991
query37	152	72	80	72
query38	2963	2867	2865	2865
query39	895	840	852	840
query40	216	139	133	133
query41	58	54	59	54
query42	115	109	111	109
query43	592	543	578	543
query44	1239	755	742	742
query45	193	169	170	169
query46	1079	728	723	723
query47	1902	1810	1788	1788
query48	380	310	304	304
query49	845	415	423	415
query50	775	385	384	384
query51	6843	6769	6703	6703
query52	108	95	93	93
query53	363	310	297	297
query54	876	452	450	450
query55	77	78	76	76
query56	301	289	274	274
query57	1124	1107	1055	1055
query58	264	268	242	242
query59	3335	3057	3087	3057
query60	307	287	289	287
query61	121	117	117	117
query62	604	454	460	454
query63	324	292	292	292
query64	8686	2353	1846	1846
query65	3200	3119	3138	3119
query66	767	345	335	335
query67	15738	15191	15133	15133
query68	4748	554	567	554
query69	725	482	395	395
query70	1179	1109	1147	1109
query71	442	272	278	272
query72	9060	5577	5265	5265
query73	758	327	325	325
query74	5905	5505	5501	5501
query75	4015	2683	2690	2683
query76	3175	919	994	919
query77	744	301	298	298
query78	10405	9925	9889	9889
query79	7729	514	522	514
query80	1143	487	464	464
query81	551	220	226	220
query82	754	105	102	102
query83	380	171	173	171
query84	268	83	84	83
query85	1297	280	264	264
query86	427	294	314	294
query87	3286	3076	3118	3076
query88	4883	2384	2409	2384
query89	480	381	384	381
query90	1863	186	182	182
query91	126	104	97	97
query92	58	50	50	50
query93	6092	529	526	526
query94	1159	188	183	183
query95	406	322	310	310
query96	608	270	274	270
query97	3243	3051	3086	3051
query98	219	203	191	191
query99	1145	861	833	833
Total cold run time: 281760 ms
Total hot run time: 175070 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99aaa3938997c35d8d9a8094960b50cd737e762b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.10	0.08
query5	0.50	0.49	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.05	0.05
query9	0.55	0.51	0.49
query10	0.53	0.53	0.53
query11	0.15	0.11	0.12
query12	0.15	0.13	0.12
query13	0.59	0.58	0.59
query14	0.78	0.78	0.78
query15	0.83	0.82	0.81
query16	0.36	0.36	0.37
query17	1.00	1.00	1.03
query18	0.22	0.24	0.23
query19	1.82	1.72	1.69
query20	0.02	0.01	0.01
query21	15.45	0.73	0.65
query22	4.29	7.08	2.26
query23	18.27	1.39	1.23
query24	2.08	0.25	0.22
query25	0.15	0.08	0.08
query26	0.26	0.17	0.17
query27	0.08	0.08	0.09
query28	13.26	1.01	1.00
query29	12.62	3.32	3.33
query30	0.26	0.06	0.06
query31	2.87	0.39	0.39
query32	3.26	0.46	0.48
query33	2.93	2.90	2.92
query34	17.18	4.39	4.39
query35	4.48	4.47	4.48
query36	0.66	0.45	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.03
query40	0.17	0.14	0.15
query41	0.09	0.04	0.05
query42	0.06	0.05	0.05
query43	0.05	0.03	0.04
Total cold run time: 109.56 s
Total hot run time: 30.89 s

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39946 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 708e7cbe5d6a9b47cca1092bc86187d0d37b3446, data reload: false

------ Round 1 ----------------------------------
q1	17969	4497	4380	4380
q2	2809	192	202	192
q3	11326	1273	1073	1073
q4	11209	794	830	794
q5	7688	2659	2678	2659
q6	226	141	142	141
q7	1036	637	603	603
q8	9566	2051	2100	2051
q9	8868	6464	6486	6464
q10	8950	3781	3684	3684
q11	481	230	238	230
q12	427	232	225	225
q13	18710	2982	2962	2962
q14	257	216	227	216
q15	518	482	481	481
q16	524	378	373	373
q17	967	693	668	668
q18	8037	7499	7394	7394
q19	4902	1437	1478	1437
q20	638	318	320	318
q21	4888	3261	3912	3261
q22	399	340	343	340
Total cold run time: 120395 ms
Total hot run time: 39946 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4386	4238	4224	4224
q2	375	263	269	263
q3	2971	2761	2730	2730
q4	1880	1610	1631	1610
q5	5241	5246	5254	5246
q6	222	130	128	128
q7	2132	1741	1759	1741
q8	3273	3342	3346	3342
q9	8324	8297	8322	8297
q10	3843	3704	3665	3665
q11	599	488	513	488
q12	782	596	593	593
q13	16274	2981	2965	2965
q14	289	258	258	258
q15	532	477	478	477
q16	460	414	424	414
q17	1790	1470	1458	1458
q18	7680	7514	7473	7473
q19	3773	1651	1571	1571
q20	1995	1782	1773	1773
q21	4996	4825	4634	4634
q22	602	556	559	556
Total cold run time: 72419 ms
Total hot run time: 53906 ms

@amorynan
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40324 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a22fa635c5ace133319b58fd12ef6a1a65ee5c8b, data reload: false

------ Round 1 ----------------------------------
q1	17699	4462	4294	4294
q2	2031	212	214	212
q3	10516	1284	1203	1203
q4	10195	845	817	817
q5	7508	2676	2660	2660
q6	220	138	142	138
q7	963	601	612	601
q8	9230	2095	2090	2090
q9	8868	6516	6541	6516
q10	9005	3767	3723	3723
q11	458	245	238	238
q12	415	236	236	236
q13	17762	3035	3001	3001
q14	267	226	232	226
q15	533	491	498	491
q16	540	397	383	383
q17	994	686	667	667
q18	8085	7493	7498	7493
q19	6181	1464	1438	1438
q20	667	318	320	318
q21	5016	3232	3985	3232
q22	426	349	347	347
Total cold run time: 117579 ms
Total hot run time: 40324 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4279	4234	4234
q2	378	263	264	263
q3	3011	2790	2708	2708
q4	1866	1598	1657	1598
q5	5244	5248	5347	5248
q6	226	133	132	132
q7	2098	1738	1732	1732
q8	3217	3371	3353	3353
q9	8348	8290	8321	8290
q10	3880	3650	3670	3650
q11	589	484	482	482
q12	778	643	599	599
q13	17345	3012	3013	3012
q14	308	263	254	254
q15	530	475	472	472
q16	467	431	413	413
q17	1817	1471	1486	1471
q18	7798	7534	7433	7433
q19	1696	1610	1594	1594
q20	1971	1799	1772	1772
q21	4954	4802	4803	4802
q22	611	597	529	529
Total cold run time: 71487 ms
Total hot run time: 54041 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169919 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a22fa635c5ace133319b58fd12ef6a1a65ee5c8b, data reload: false

query1	927	376	370	370
query2	6470	2569	2429	2429
query3	6646	208	215	208
query4	21161	17499	17343	17343
query5	4186	487	472	472
query6	267	174	177	174
query7	4593	295	305	295
query8	312	299	293	293
query9	8557	2355	2341	2341
query10	610	306	290	290
query11	10523	10103	10067	10067
query12	139	88	86	86
query13	1670	365	370	365
query14	10358	7286	6279	6279
query15	236	187	197	187
query16	7757	271	276	271
query17	1897	567	542	542
query18	1914	281	284	281
query19	197	156	160	156
query20	91	90	87	87
query21	221	130	134	130
query22	4277	4079	4007	4007
query23	33696	32946	33062	32946
query24	10983	2839	2840	2839
query25	660	379	379	379
query26	1554	157	159	157
query27	2952	321	318	318
query28	7482	2041	2049	2041
query29	993	644	633	633
query30	276	161	158	158
query31	981	745	755	745
query32	91	55	58	55
query33	796	299	304	299
query34	955	480	485	480
query35	754	663	646	646
query36	1134	940	961	940
query37	167	76	72	72
query38	2900	2730	2773	2730
query39	871	828	834	828
query40	230	131	131	131
query41	58	55	54	54
query42	126	102	102	102
query43	601	528	563	528
query44	1224	746	739	739
query45	201	162	169	162
query46	1071	727	705	705
query47	1838	1783	1762	1762
query48	377	303	309	303
query49	1163	433	437	433
query50	785	397	408	397
query51	6901	6757	6779	6757
query52	110	96	99	96
query53	364	301	305	301
query54	1029	460	470	460
query55	77	77	76	76
query56	310	287	285	285
query57	1154	1083	1034	1034
query58	260	260	262	260
query59	3585	3296	3250	3250
query60	317	293	293	293
query61	98	110	94	94
query62	669	440	452	440
query63	318	291	289	289
query64	9993	2215	1778	1778
query65	3173	3065	3076	3065
query66	1372	340	329	329
query67	15760	14932	14981	14932
query68	8117	577	563	563
query69	700	473	430	430
query70	1194	1213	1095	1095
query71	500	276	279	276
query72	9215	2761	2583	2583
query73	821	330	326	326
query74	5968	5391	5513	5391
query75	5141	2596	2681	2596
query76	4720	974	936	936
query77	780	301	307	301
query78	10463	9758	9684	9684
query79	7976	518	509	509
query80	998	469	474	469
query81	542	219	223	219
query82	776	110	105	105
query83	424	178	171	171
query84	268	87	89	87
query85	1388	272	312	272
query86	396	324	300	300
query87	3258	3102	3097	3097
query88	4787	2454	2459	2454
query89	543	383	384	383
query90	2007	194	191	191
query91	134	101	102	101
query92	65	49	50	49
query93	6535	498	511	498
query94	1240	200	192	192
query95	415	315	316	315
query96	603	276	264	264
query97	3209	3039	3056	3039
query98	215	201	194	194
query99	1214	852	835	835
Total cold run time: 298030 ms
Total hot run time: 169919 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a22fa635c5ace133319b58fd12ef6a1a65ee5c8b, data reload: false

query1	0.04	0.04	0.02
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.10	0.09
query5	0.50	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.05	0.04
query9	0.56	0.49	0.48
query10	0.55	0.54	0.53
query11	0.15	0.11	0.12
query12	0.16	0.12	0.12
query13	0.58	0.59	0.60
query14	0.77	0.79	0.79
query15	0.82	0.82	0.82
query16	0.36	0.36	0.35
query17	0.99	0.97	0.96
query18	0.22	0.23	0.24
query19	1.85	1.75	1.74
query20	0.01	0.01	0.02
query21	15.46	0.74	0.63
query22	5.11	6.59	1.61
query23	18.24	1.39	1.32
query24	2.06	0.23	0.22
query25	0.15	0.08	0.08
query26	0.27	0.17	0.18
query27	0.07	0.08	0.08
query28	13.22	1.03	0.99
query29	12.59	3.24	3.28
query30	0.25	0.06	0.05
query31	2.85	0.39	0.39
query32	3.28	0.46	0.48
query33	2.87	2.90	2.90
query34	17.06	4.43	4.40
query35	4.46	4.46	4.54
query36	0.65	0.47	0.46
query37	0.17	0.15	0.15
query38	0.16	0.14	0.14
query39	0.05	0.04	0.03
query40	0.17	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.06 s
Total hot run time: 30.24 s

morrySnow
morrySnow previously approved these changes Jun 27, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jun 27, 2024
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jun 28, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40614 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 34058b9ed3212e5164b8797302bfa094d92a5227, data reload: false

------ Round 1 ----------------------------------
q1	18154	4434	4338	4338
q2	2008	188	191	188
q3	10493	1142	1120	1120
q4	10255	820	809	809
q5	8642	2711	2641	2641
q6	216	138	137	137
q7	968	604	613	604
q8	9226	2086	2062	2062
q9	9107	6608	6550	6550
q10	9269	3912	3902	3902
q11	451	246	249	246
q12	493	246	254	246
q13	17917	3245	3169	3169
q14	284	232	236	232
q15	527	512	488	488
q16	663	399	391	391
q17	1053	723	720	720
q18	8586	7503	7418	7418
q19	7958	1546	1571	1546
q20	668	320	318	318
q21	4990	3156	3290	3156
q22	393	341	333	333
Total cold run time: 122321 ms
Total hot run time: 40614 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4380	4205	4246	4205
q2	376	275	281	275
q3	3083	2916	2908	2908
q4	1995	1699	1667	1667
q5	5491	5429	5581	5429
q6	221	132	133	132
q7	2206	1855	1847	1847
q8	3261	3381	3377	3377
q9	8785	8772	8753	8753
q10	3956	3819	3845	3819
q11	578	495	506	495
q12	805	650	625	625
q13	17242	3209	3197	3197
q14	298	304	269	269
q15	526	491	479	479
q16	509	441	439	439
q17	1808	1518	1494	1494
q18	8178	7786	7774	7774
q19	1867	1724	1573	1573
q20	3221	1871	1898	1871
q21	4972	4820	4875	4820
q22	729	546	543	543
Total cold run time: 74487 ms
Total hot run time: 55991 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174471 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 34058b9ed3212e5164b8797302bfa094d92a5227, data reload: false

query1	918	382	367	367
query2	6370	2659	2322	2322
query3	6638	206	215	206
query4	18658	17618	17340	17340
query5	3662	510	502	502
query6	266	176	181	176
query7	4599	301	325	301
query8	335	309	327	309
query9	8638	2392	2384	2384
query10	560	307	282	282
query11	10650	9997	9918	9918
query12	113	86	88	86
query13	1642	370	356	356
query14	9772	7527	7857	7527
query15	241	194	186	186
query16	7828	274	265	265
query17	1766	558	556	556
query18	1950	265	270	265
query19	201	155	152	152
query20	91	91	94	91
query21	212	135	123	123
query22	4495	4067	4154	4067
query23	33581	33567	33741	33567
query24	11226	2948	2899	2899
query25	602	400	375	375
query26	737	163	155	155
query27	2367	326	335	326
query28	6187	2138	2163	2138
query29	903	625	669	625
query30	257	158	159	158
query31	996	769	760	760
query32	100	53	60	53
query33	817	288	288	288
query34	1028	495	499	495
query35	774	633	637	633
query36	1161	1013	991	991
query37	150	73	75	73
query38	2945	2829	2840	2829
query39	905	845	841	841
query40	218	135	133	133
query41	58	54	57	54
query42	108	108	102	102
query43	648	617	563	563
query44	1276	738	724	724
query45	198	160	158	158
query46	1066	733	741	733
query47	1854	1798	1747	1747
query48	378	306	295	295
query49	845	411	413	411
query50	771	388	394	388
query51	6906	6900	6731	6731
query52	107	94	91	91
query53	364	301	288	288
query54	871	446	439	439
query55	76	76	73	73
query56	298	276	284	276
query57	1115	1034	1060	1034
query58	251	253	244	244
query59	3448	3409	3301	3301
query60	309	278	288	278
query61	96	90	91	90
query62	593	459	450	450
query63	319	306	300	300
query64	8648	2228	1764	1764
query65	3185	3098	3095	3095
query66	764	340	328	328
query67	15636	15113	15110	15110
query68	4531	548	540	540
query69	558	471	386	386
query70	1217	1082	1181	1082
query71	371	287	279	279
query72	7296	5477	4938	4938
query73	756	328	330	328
query74	5854	5438	5511	5438
query75	3336	2690	2647	2647
query76	2279	946	887	887
query77	442	306	304	304
query78	10440	9854	9752	9752
query79	2241	525	526	525
query80	1255	538	495	495
query81	599	233	221	221
query82	771	109	107	107
query83	266	173	168	168
query84	233	86	91	86
query85	1625	285	276	276
query86	459	309	328	309
query87	3213	3112	3065	3065
query88	4010	2366	2359	2359
query89	478	398	389	389
query90	1723	195	200	195
query91	132	99	107	99
query92	58	50	52	50
query93	2366	524	533	524
query94	1084	204	194	194
query95	405	327	322	322
query96	609	271	262	262
query97	3298	3026	3051	3026
query98	221	202	202	202
query99	1318	856	829	829
Total cold run time: 268191 ms
Total hot run time: 174471 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.76 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 34058b9ed3212e5164b8797302bfa094d92a5227, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.09	0.08
query5	0.49	0.49	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.54	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.78	0.78	0.77
query15	0.86	0.82	0.81
query16	0.37	0.36	0.37
query17	0.98	1.04	0.97
query18	0.20	0.26	0.24
query19	1.79	1.69	1.78
query20	0.02	0.01	0.01
query21	15.46	0.75	0.66
query22	4.47	6.62	2.08
query23	18.30	1.36	1.29
query24	2.07	0.23	0.22
query25	0.16	0.10	0.09
query26	0.28	0.18	0.17
query27	0.08	0.08	0.08
query28	13.29	1.02	0.98
query29	12.59	3.33	3.38
query30	0.25	0.06	0.07
query31	2.88	0.38	0.38
query32	3.26	0.46	0.47
query33	2.89	2.96	2.88
query34	17.25	4.43	4.42
query35	4.46	4.44	4.50
query36	0.66	0.47	0.49
query37	0.18	0.17	0.16
query38	0.17	0.14	0.15
query39	0.05	0.04	0.04
query40	0.18	0.15	0.15
query41	0.09	0.04	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 30.76 s

@amorynan amorynan requested a review from morrySnow July 1, 2024 11:36
morrySnow
morrySnow previously approved these changes Jul 1, 2024
@amorynan amorynan dismissed stale reviews from morrySnow and eldenmoon via 1c43cd3 July 1, 2024 14:32
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 1, 2024
@amorynan
Copy link
Contributor Author

amorynan commented Jul 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40066 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ca859676c5c52431759188cefae11a74e9207d38, data reload: false

------ Round 1 ----------------------------------
q1	17602	4567	4288	4288
q2	2018	193	191	191
q3	10444	1253	1123	1123
q4	10197	823	811	811
q5	7530	2691	2671	2671
q6	227	143	140	140
q7	958	604	614	604
q8	9228	2104	2099	2099
q9	9289	6507	6517	6507
q10	8948	3717	3697	3697
q11	455	242	238	238
q12	459	239	233	233
q13	17927	2964	2987	2964
q14	270	228	228	228
q15	530	492	495	492
q16	497	382	376	376
q17	980	659	742	659
q18	7946	7398	7452	7398
q19	4723	1469	1474	1469
q20	642	317	331	317
q21	4897	3229	3939	3229
q22	402	332	334	332
Total cold run time: 116169 ms
Total hot run time: 40066 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4370	4238	4259	4238
q2	382	277	263	263
q3	3015	2851	2832	2832
q4	1991	1743	1631	1631
q5	5710	5490	5520	5490
q6	220	131	132	131
q7	2208	1884	1877	1877
q8	3266	3404	3464	3404
q9	8712	8738	8900	8738
q10	4161	3847	3720	3720
q11	597	498	474	474
q12	852	632	625	625
q13	16116	3247	3185	3185
q14	302	276	276	276
q15	547	468	504	468
q16	500	445	438	438
q17	1806	1561	1497	1497
q18	8048	7962	7905	7905
q19	1850	1656	1652	1652
q20	2878	1885	1868	1868
q21	8918	5016	4676	4676
q22	633	554	570	554
Total cold run time: 77082 ms
Total hot run time: 55942 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173411 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ca859676c5c52431759188cefae11a74e9207d38, data reload: false

query1	915	398	380	380
query2	6484	2588	2447	2447
query3	6647	217	218	217
query4	19136	17598	17224	17224
query5	3666	491	514	491
query6	273	186	163	163
query7	4585	308	290	290
query8	324	298	297	297
query9	8587	2371	2352	2352
query10	571	312	287	287
query11	10574	10034	9990	9990
query12	114	85	86	85
query13	1640	348	358	348
query14	10300	6288	7391	6288
query15	234	186	188	186
query16	7765	263	262	262
query17	1477	565	515	515
query18	1948	274	268	268
query19	192	155	152	152
query20	89	82	82	82
query21	209	131	123	123
query22	4417	3960	4019	3960
query23	33643	33674	33641	33641
query24	10950	2817	2848	2817
query25	611	388	392	388
query26	730	155	152	152
query27	2334	337	323	323
query28	5957	2145	2135	2135
query29	901	703	628	628
query30	264	157	155	155
query31	1004	780	757	757
query32	99	57	59	57
query33	685	299	293	293
query34	875	493	494	493
query35	763	639	646	639
query36	1088	958	980	958
query37	138	83	79	79
query38	2967	2817	2802	2802
query39	910	838	836	836
query40	227	135	130	130
query41	55	53	54	53
query42	115	104	109	104
query43	602	558	574	558
query44	1103	752	732	732
query45	203	170	168	168
query46	1082	745	712	712
query47	1840	1746	1749	1746
query48	366	312	310	310
query49	872	439	436	436
query50	777	397	397	397
query51	6868	6844	6779	6779
query52	97	101	93	93
query53	364	303	305	303
query54	914	454	450	450
query55	75	78	75	75
query56	319	293	304	293
query57	1142	1029	1019	1019
query58	272	265	261	261
query59	3567	3243	3175	3175
query60	335	300	301	300
query61	117	117	113	113
query62	625	448	449	448
query63	336	308	302	302
query64	8850	2356	1869	1869
query65	3154	3116	3118	3116
query66	759	340	344	340
query67	15623	14920	15044	14920
query68	6235	551	556	551
query69	720	469	363	363
query70	1202	1091	1103	1091
query71	460	293	293	293
query72	8815	5414	5382	5382
query73	774	330	327	327
query74	5849	5469	5488	5469
query75	4641	2652	2653	2652
query76	3705	924	947	924
query77	815	317	307	307
query78	10321	9796	9825	9796
query79	8219	520	502	502
query80	1049	471	476	471
query81	557	220	215	215
query82	785	103	104	103
query83	329	170	163	163
query84	270	88	87	87
query85	1250	269	303	269
query86	405	279	327	279
query87	3254	3083	3039	3039
query88	4507	2375	2378	2375
query89	532	387	389	387
query90	1908	185	185	185
query91	128	97	101	97
query92	59	50	53	50
query93	6156	499	498	498
query94	1188	191	187	187
query95	405	314	320	314
query96	594	267	267	267
query97	3195	2967	3019	2967
query98	211	209	195	195
query99	1131	860	844	844
Total cold run time: 284394 ms
Total hot run time: 173411 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.4 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ca859676c5c52431759188cefae11a74e9207d38, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.06
query5	0.50	0.48	0.49
query6	1.13	0.72	0.74
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.54	0.49	0.50
query10	0.53	0.55	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.13
query13	0.59	0.58	0.59
query14	0.76	0.77	0.78
query15	0.85	0.81	0.82
query16	0.37	0.37	0.36
query17	1.03	1.02	1.00
query18	0.23	0.22	0.26
query19	1.92	1.77	1.73
query20	0.02	0.01	0.02
query21	15.47	0.79	0.66
query22	4.23	7.98	1.71
query23	18.24	1.49	1.30
query24	2.17	0.23	0.23
query25	0.16	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.25	1.02	1.00
query29	12.58	3.28	3.28
query30	0.25	0.06	0.06
query31	2.86	0.39	0.39
query32	3.26	0.48	0.48
query33	2.83	2.91	2.90
query34	17.14	4.45	4.36
query35	4.44	4.46	4.49
query36	0.65	0.47	0.50
query37	0.18	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.48 s
Total hot run time: 30.4 s

@amorynan amorynan requested a review from eldenmoon July 2, 2024 07:37
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit d60dbb5 into apache:master Jul 2, 2024
25 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
after this pr we support
expand json object into two column with key column (string) ,and value
column(json value)
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants