Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](jni_connector) fix that be core when jni_connector open fails #37697

Merged
merged 2 commits into from
Jul 14, 2024
Merged

[fix](jni_connector) fix that be core when jni_connector open fails #37697

merged 2 commits into from
Jul 14, 2024

Conversation

BePPPower
Copy link
Contributor

@BePPPower BePPPower commented Jul 12, 2024 •
edited
Loading

Proposed changes

Here are two problems:

  1. appendDataTimeNs will be NULL if TrinoConnector open fails before call to parseRequiredTypes
  2. It does not catch exception in method get_statistics() and this exception is not caught until close() method. Therefore, it will cause BE core dump in JniConnector::close() at LOG(FATAL)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40159 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bb50f2d90af34154aec791597ffb79403b65b761, data reload: false

------ Round 1 ----------------------------------
q1	17622	4317	4297	4297
q2	2023	195	188	188
q3	10430	1193	1103	1103
q4	10417	828	779	779
q5	7553	2693	2724	2693
q6	222	142	139	139
q7	961	609	607	607
q8	9260	2091	2094	2091
q9	9016	6647	6598	6598
q10	8855	3853	3830	3830
q11	445	236	241	236
q12	408	228	231	228
q13	19202	3010	3027	3010
q14	279	244	247	244
q15	530	500	502	500
q16	499	389	383	383
q17	969	673	755	673
q18	8132	7501	7343	7343
q19	8451	1342	1390	1342
q20	714	356	341	341
q21	5235	3193	3382	3193
q22	394	344	341	341
Total cold run time: 121617 ms
Total hot run time: 40159 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4599	4436	4399	4399
q2	383	291	288	288
q3	3333	3135	3120	3120
q4	2045	1784	1679	1679
q5	5598	5544	5659	5544
q6	238	148	149	148
q7	2282	1829	1883	1829
q8	3272	3380	3390	3380
q9	8779	8934	8735	8735
q10	3986	3776	3894	3776
q11	590	499	496	496
q12	821	657	628	628
q13	16250	3225	3149	3149
q14	313	313	292	292
q15	515	504	492	492
q16	475	434	422	422
q17	1796	1537	1512	1512
q18	8072	8023	7885	7885
q19	1792	1536	1518	1518
q20	2695	1889	1841	1841
q21	11889	4773	4765	4765
q22	628	571	554	554
Total cold run time: 80351 ms
Total hot run time: 56452 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175289 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bb50f2d90af34154aec791597ffb79403b65b761, data reload: false

query1	911	383	360	360
query2	6434	2449	2331	2331
query3	6643	210	222	210
query4	28552	17300	17338	17300
query5	3664	499	475	475
query6	269	183	172	172
query7	4599	295	297	295
query8	303	293	283	283
query9	8459	2459	2450	2450
query10	447	281	274	274
query11	10459	9980	10060	9980
query12	118	87	80	80
query13	1637	378	372	372
query14	10077	7673	8222	7673
query15	255	189	184	184
query16	7444	312	310	310
query17	1809	546	536	536
query18	1851	272	271	271
query19	196	148	148	148
query20	90	79	83	79
query21	204	124	124	124
query22	4323	4258	4109	4109
query23	33970	33682	33528	33528
query24	11112	2919	2874	2874
query25	607	403	386	386
query26	706	152	152	152
query27	2232	281	283	281
query28	6444	2188	2195	2188
query29	902	637	674	637
query30	258	152	155	152
query31	968	763	773	763
query32	99	56	55	55
query33	776	305	312	305
query34	990	496	501	496
query35	710	604	569	569
query36	1129	1010	994	994
query37	140	83	90	83
query38	2923	2863	2831	2831
query39	892	865	806	806
query40	197	123	120	120
query41	52	49	50	49
query42	116	99	99	99
query43	582	566	538	538
query44	1226	730	740	730
query45	194	168	168	168
query46	1100	750	720	720
query47	1875	1789	1774	1774
query48	375	297	296	296
query49	839	428	419	419
query50	786	393	398	393
query51	6881	6770	6800	6770
query52	103	97	93	93
query53	361	287	297	287
query54	876	457	455	455
query55	80	76	75	75
query56	292	270	267	267
query57	1105	1035	1076	1035
query58	259	250	262	250
query59	3402	3242	3103	3103
query60	306	282	275	275
query61	97	97	93	93
query62	821	662	631	631
query63	318	287	289	287
query64	9138	2181	1671	1671
query65	3161	3128	3090	3090
query66	746	329	333	329
query67	15406	15009	15004	15004
query68	5669	561	566	561
query69	594	447	377	377
query70	1126	1148	1155	1148
query71	455	277	272	272
query72	7756	5869	5929	5869
query73	773	328	335	328
query74	5978	5568	5512	5512
query75	3574	2714	2698	2698
query76	3741	992	872	872
query77	622	309	325	309
query78	10132	8949	8998	8949
query79	2424	522	529	522
query80	2097	483	482	482
query81	587	221	222	221
query82	1384	139	133	133
query83	323	173	172	172
query84	279	91	89	89
query85	1353	302	297	297
query86	478	305	332	305
query87	3294	3128	3078	3078
query88	3663	2452	2538	2452
query89	471	384	394	384
query90	1776	196	193	193
query91	129	103	99	99
query92	63	50	48	48
query93	2318	529	523	523
query94	1100	210	211	210
query95	407	322	323	322
query96	601	277	279	277
query97	3203	3017	3005	3005
query98	235	202	197	197
query99	1685	1282	1248	1248
Total cold run time: 282586 ms
Total hot run time: 175289 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.79 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bb50f2d90af34154aec791597ffb79403b65b761, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.09	0.07
query5	0.49	0.49	0.47
query6	1.13	0.72	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.47	0.50
query10	0.56	0.54	0.52
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.59	0.58	0.58
query14	0.79	0.75	0.79
query15	0.85	0.80	0.80
query16	0.36	0.36	0.37
query17	1.04	0.96	1.02
query18	0.22	0.22	0.21
query19	1.75	1.74	1.70
query20	0.01	0.01	0.01
query21	15.40	0.75	0.65
query22	4.15	7.18	2.33
query23	18.29	1.41	1.19
query24	2.06	0.23	0.22
query25	0.16	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.24	0.22
query28	13.26	1.01	1.01
query29	12.67	3.28	3.28
query30	0.25	0.06	0.06
query31	2.89	0.38	0.39
query32	3.26	0.47	0.46
query33	2.91	2.97	2.91
query34	17.06	4.33	4.36
query35	4.42	4.48	4.41
query36	0.65	0.46	0.47
query37	0.19	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.09	0.04	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.61 s
Total hot run time: 30.79 s

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@BePPPower
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40266 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2fc0fd7e7e0af5028b1257efe40c58a5760e129e, data reload: false

------ Round 1 ----------------------------------
q1	17988	4507	4381	4381
q2	2026	191	186	186
q3	10461	1255	1168	1168
q4	10193	838	775	775
q5	7556	2750	2725	2725
q6	226	136	138	136
q7	974	600	607	600
q8	9237	2096	2073	2073
q9	8660	6563	6570	6563
q10	8631	3803	3765	3765
q11	471	241	247	241
q12	404	236	233	233
q13	18887	2983	3004	2983
q14	276	243	231	231
q15	525	483	479	479
q16	496	396	385	385
q17	987	619	724	619
q18	8066	7586	7427	7427
q19	6698	1528	1518	1518
q20	708	331	341	331
q21	4976	3166	3195	3166
q22	347	291	281	281
Total cold run time: 118793 ms
Total hot run time: 40266 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4389	4282	4258	4258
q2	375	285	272	272
q3	3018	2792	2747	2747
q4	1895	1656	1618	1618
q5	5345	5340	5299	5299
q6	216	131	131	131
q7	2145	1702	1743	1702
q8	3217	3349	3314	3314
q9	8430	8428	8433	8428
q10	3920	3687	3721	3687
q11	592	486	498	486
q12	783	637	606	606
q13	16512	3018	2977	2977
q14	314	275	264	264
q15	514	469	474	469
q16	465	406	414	406
q17	1759	1494	1480	1480
q18	7748	7490	7384	7384
q19	1971	1474	1614	1474
q20	2018	1792	1781	1781
q21	4780	4796	4643	4643
q22	556	484	498	484
Total cold run time: 70962 ms
Total hot run time: 53910 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172786 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2fc0fd7e7e0af5028b1257efe40c58a5760e129e, data reload: false

query1	906	386	367	367
query2	6453	2016	1855	1855
query3	6670	205	219	205
query4	27449	17440	17374	17374
query5	4225	506	487	487
query6	272	178	168	168
query7	4595	288	298	288
query8	245	203	205	203
query9	8606	2442	2432	2432
query10	432	304	296	296
query11	11612	10275	10061	10061
query12	142	86	88	86
query13	1657	382	373	373
query14	10264	7203	7041	7041
query15	218	173	173	173
query16	7826	323	325	323
query17	1557	554	551	551
query18	1973	293	287	287
query19	203	159	161	159
query20	87	83	87	83
query21	206	134	126	126
query22	4399	4052	4136	4052
query23	33864	33137	33025	33025
query24	12077	2926	2921	2921
query25	680	389	402	389
query26	1830	154	153	153
query27	2932	278	277	277
query28	7780	2041	2049	2041
query29	1188	654	635	635
query30	286	151	149	149
query31	938	765	779	765
query32	101	58	64	58
query33	791	327	323	323
query34	905	506	496	496
query35	707	612	588	588
query36	1115	948	982	948
query37	197	83	87	83
query38	2912	2763	2734	2734
query39	875	820	816	816
query40	281	124	126	124
query41	47	46	48	46
query42	117	101	102	101
query43	498	484	467	467
query44	1261	773	751	751
query45	192	162	159	159
query46	1091	754	750	750
query47	1860	1751	1805	1751
query48	385	304	299	299
query49	1203	443	431	431
query50	796	413	415	413
query51	6963	6913	6827	6827
query52	103	96	101	96
query53	375	304	296	296
query54	957	464	458	458
query55	79	77	76	76
query56	304	300	299	299
query57	1149	1029	1042	1029
query58	273	252	373	252
query59	2870	2728	2631	2631
query60	315	274	279	274
query61	98	98	100	98
query62	818	644	674	644
query63	337	296	292	292
query64	10446	2197	1689	1689
query65	3173	3169	3116	3116
query66	1378	337	356	337
query67	15673	15032	14934	14934
query68	9181	575	569	569
query69	836	434	385	385
query70	1426	1108	1122	1108
query71	547	285	281	281
query72	9033	5557	5288	5288
query73	2256	333	330	330
query74	6147	5608	5500	5500
query75	5720	2672	2714	2672
query76	5552	1044	964	964
query77	773	328	324	324
query78	9903	9075	8957	8957
query79	11344	546	557	546
query80	834	507	483	483
query81	580	227	224	224
query82	301	132	129	129
query83	340	166	166	166
query84	274	85	87	85
query85	1029	306	355	306
query86	359	318	307	307
query87	3340	3079	3131	3079
query88	4787	2470	2504	2470
query89	503	386	379	379
query90	2083	200	202	200
query91	134	100	104	100
query92	68	50	50	50
query93	5458	527	525	525
query94	1502	222	220	220
query95	406	324	330	324
query96	637	280	279	279
query97	3212	3048	3041	3041
query98	217	198	190	190
query99	1538	1301	1274	1274
Total cold run time: 312537 ms
Total hot run time: 172786 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2fc0fd7e7e0af5028b1257efe40c58a5760e129e, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.04
query4	1.68	0.06	0.08
query5	0.49	0.48	0.50
query6	1.12	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.56	0.54	0.55
query11	0.16	0.12	0.12
query12	0.14	0.11	0.12
query13	0.60	0.58	0.58
query14	0.74	0.77	0.81
query15	0.84	0.81	0.81
query16	0.37	0.36	0.36
query17	1.00	1.04	0.98
query18	0.23	0.22	0.23
query19	1.82	1.69	1.66
query20	0.01	0.02	0.01
query21	15.43	0.78	0.66
query22	4.40	6.40	2.34
query23	18.34	1.40	1.31
query24	2.09	0.23	0.22
query25	0.15	0.09	0.08
query26	0.29	0.21	0.22
query27	0.46	0.24	0.23
query28	13.30	1.02	0.98
query29	12.64	3.28	3.25
query30	0.26	0.06	0.06
query31	2.90	0.38	0.38
query32	3.28	0.49	0.46
query33	2.85	2.87	2.84
query34	17.10	4.30	4.36
query35	4.42	4.43	4.40
query36	0.65	0.46	0.48
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.11	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.05	0.04
Total cold run time: 110.02 s
Total hot run time: 30.82 s

morningman
morningman approved these changes Jul 12, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 12, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 2f2ccc6 into apache:master Jul 14, 2024
26 of 30 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@BePPPower @seawinde
[fix](jni_connector) fix that be core when jni_connector open fails (a…
4c6132c 
…pache#37697)

Here are two problems:
1. `appendDataTimeNs` will be `NULL` if TrinoConnector open fails before
call to `parseRequiredTypes`
2. It does not catch exception in method `get_statistics()` and this
exception is not caught until `close()` method. Therefore, it will cause
BE core dump in `JniConnector::close()` at `LOG(FATAL)`
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@BePPPower @dataroaring
[fix](jni_connector) fix that be core when jni_connector open fails (#…
18bc6af 
…37697)

Here are two problems:
1. `appendDataTimeNs` will be `NULL` if TrinoConnector open fails before
call to `parseRequiredTypes`
2. It does not catch exception in method `get_statistics()` and this
exception is not caught until `close()` method. Therefore, it will cause
BE core dump in `JniConnector::close()` at `LOG(FATAL)`
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hello-stephen hello-stephen hello-stephen approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@BePPPower @doris-robot @morningman @hello-stephen @yiguolei @dataroaring