Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](JsonReader) fix json with duplicate key entry may result out of bound exception #38146

Merged
merged 1 commit into from
Jul 22, 2024
Merged

[Fix](JsonReader) fix json with duplicate key entry may result out of bound exception #38146

merged 1 commit into from
Jul 22, 2024

Conversation

eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jul 19, 2024 •
edited
Loading

No description provided.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@eldenmoon eldenmoon changed the title [Fix](JsonReader) fix json with duplicate key entry may result out of… [Fix](JsonReader) fix json with duplicate key entry may result out of bound exception Jul 19, 2024
@eldenmoon
Copy link
Member Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

2 similar comments
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@eldenmoon eldenmoon mentioned this pull request Jul 19, 2024
Merged
@doris-robot
Copy link

TPC-H: Total hot run time: 39946 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b32222e50990e2aee7faf8d04dc096f28df6f6bf, data reload: false

------ Round 1 ----------------------------------
q1	19595	4513	4394	4394
q2	2646	199	186	186
q3	11686	1207	1059	1059
q4	10239	849	867	849
q5	7614	2706	2668	2668
q6	224	142	138	138
q7	979	602	607	602
q8	9227	2081	2058	2058
q9	8709	6566	6632	6566
q10	8798	3804	3772	3772
q11	452	234	239	234
q12	396	233	223	223
q13	17761	2970	2993	2970
q14	271	242	240	240
q15	541	482	478	478
q16	502	379	388	379
q17	965	714	704	704
q18	8122	7501	7336	7336
q19	8568	1380	1525	1380
q20	711	310	339	310
q21	4897	3119	3158	3119
q22	343	293	281	281
Total cold run time: 123246 ms
Total hot run time: 39946 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4367	4263	4263
q2	393	269	264	264
q3	3001	2772	2732	2732
q4	1859	1583	1561	1561
q5	5303	5313	5331	5313
q6	222	131	131	131
q7	2120	1750	1710	1710
q8	3223	3333	3311	3311
q9	8397	8429	8391	8391
q10	3889	3702	3733	3702
q11	581	469	478	469
q12	772	624	621	621
q13	16362	2955	2965	2955
q14	311	269	273	269
q15	516	499	474	474
q16	474	409	406	406
q17	1774	1493	1465	1465
q18	7730	7530	7355	7355
q19	1653	1640	1379	1379
q20	1966	1781	1791	1781
q21	4852	4741	4807	4741
q22	591	503	519	503
Total cold run time: 70391 ms
Total hot run time: 53796 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173436 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b32222e50990e2aee7faf8d04dc096f28df6f6bf, data reload: false

query1	939	369	364	364
query2	6459	1835	1789	1789
query3	6674	211	219	211
query4	28408	17512	17362	17362
query5	4231	474	474	474
query6	277	180	168	168
query7	4591	287	284	284
query8	257	201	203	201
query9	8554	2425	2416	2416
query10	458	295	276	276
query11	10642	10119	10103	10103
query12	137	90	84	84
query13	1649	367	375	367
query14	10274	7899	7495	7495
query15	217	168	167	167
query16	7630	505	443	443
query17	1584	569	553	553
query18	1448	295	300	295
query19	206	155	152	152
query20	92	87	87	87
query21	209	125	132	125
query22	4302	4123	4048	4048
query23	33847	33395	33078	33078
query24	12151	2940	2912	2912
query25	669	404	401	401
query26	1751	152	155	152
query27	2844	270	280	270
query28	7605	1997	1993	1993
query29	1057	650	635	635
query30	285	152	154	152
query31	959	769	762	762
query32	99	55	56	55
query33	784	342	338	338
query34	885	474	476	474
query35	869	739	769	739
query36	1105	973	962	962
query37	258	80	81	80
query38	2859	2810	2769	2769
query39	850	804	812	804
query40	283	125	123	123
query41	49	53	47	47
query42	130	103	106	103
query43	499	471	469	469
query44	1175	731	731	731
query45	190	160	163	160
query46	1094	750	727	727
query47	1852	1785	1827	1785
query48	383	292	307	292
query49	1191	437	432	432
query50	796	401	389	389
query51	7008	6868	6783	6783
query52	110	90	97	90
query53	368	300	285	285
query54	1013	462	454	454
query55	76	76	77	76
query56	312	298	303	298
query57	1176	1092	1051	1051
query58	276	259	280	259
query59	2991	2779	2521	2521
query60	314	275	279	275
query61	98	92	93	92
query62	815	658	668	658
query63	324	291	289	289
query64	10693	2236	1681	1681
query65	3215	3139	3106	3106
query66	1297	341	345	341
query67	16028	15141	15000	15000
query68	8867	549	567	549
query69	767	441	379	379
query70	1444	1151	1132	1132
query71	549	277	286	277
query72	8791	5102	5682	5102
query73	2171	339	327	327
query74	6133	5670	5650	5650
query75	5278	2686	2711	2686
query76	5394	1013	980	980
query77	764	317	310	310
query78	9833	9209	9493	9209
query79	11720	541	530	530
query80	1104	490	484	484
query81	584	224	223	223
query82	289	138	139	138
query83	351	164	164	164
query84	275	87	86	86
query85	1031	322	296	296
query86	340	341	286	286
query87	3347	3143	3115	3115
query88	5181	2382	2361	2361
query89	511	390	387	387
query90	1895	200	196	196
query91	130	100	100	100
query92	56	49	52	49
query93	6867	510	512	510
query94	827	299	283	283
query95	411	321	319	319
query96	635	274	270	270
query97	3192	3038	3038	3038
query98	220	206	210	206
query99	1521	1259	1237	1237
Total cold run time: 312349 ms
Total hot run time: 173436 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b32222e50990e2aee7faf8d04dc096f28df6f6bf, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.05	0.06
query4	1.68	0.07	0.07
query5	0.48	0.49	0.49
query6	1.14	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.05
query9	0.54	0.50	0.48
query10	0.55	0.56	0.56
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.75	0.78	0.78
query15	0.84	0.81	0.81
query16	0.37	0.36	0.37
query17	1.01	0.96	0.95
query18	0.23	0.21	0.22
query19	1.76	1.85	1.71
query20	0.01	0.01	0.01
query21	15.39	0.76	0.67
query22	4.19	7.19	1.91
query23	18.28	1.44	1.26
query24	2.13	0.23	0.24
query25	0.15	0.09	0.08
query26	0.32	0.21	0.22
query27	0.45	0.23	0.23
query28	13.22	1.02	1.01
query29	12.62	3.37	3.30
query30	0.26	0.06	0.06
query31	2.92	0.39	0.38
query32	3.28	0.46	0.47
query33	2.91	2.94	2.94
query34	17.08	4.54	4.41
query35	4.46	4.46	4.43
query36	0.65	0.46	0.49
query37	0.18	0.17	0.16
query38	0.15	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.68 s
Total hot run time: 30.77 s

yiguolei pushed a commit that referenced this pull request Jul 19, 2024
@eldenmoon
[Fix](JsonReader) fix json with duplicate key entry may result out of…
d9fd419 
… bound exception (#38147)

#38146
xiaokang
xiaokang approved these changes Jul 22, 2024
View reviewed changes
Copy link
Contributor

@xiaokang xiaokang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@eldenmoon eldenmoon mentioned this pull request Jul 22, 2024
Merged
@eldenmoon eldenmoon merged commit 84afb29 into apache:master Jul 22, 2024
28 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Jul 23, 2024
@eldenmoon @dataroaring
[Fix](JsonReader) fix json with duplicate key entry may result out of…
7378d06 
… bound exception (#38146)
eldenmoon added a commit that referenced this pull request Jul 23, 2024
@eldenmoon
[Fix](JsonReader) fix json with duplicate key entry may result out of…
1631076 
… bound exception (#38188)

#38146
@eldenmoon eldenmoon deleted the fix-dup-json branch July 24, 2024 17:00
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@eldenmoon
[Fix](JsonReader) fix json with duplicate key entry may result out of…
9f462c2 
… bound exception (apache#38188)

apache#38146
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@qidaye qidaye qidaye approved these changes

@xiaokang xiaokang xiaokang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.14-merged dev/2.1.5-merged dev/3.0.0-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@eldenmoon @doris-robot @xiaokang @qidaye @yiguolei @wm1581066