Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](publish) Catch exception in genPublishTask to make one failed txn does not block the other txns #37724

Merged
merged 1 commit into from
Jul 17, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Jul 12, 2024

Proposed changes

If any exception(such as NullPointerException) is thrown in genPublishTask when publish, the publish for all txns will fail.
This pr catch the exception to make the failed txn does not block other txns.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 12, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39592 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e15da6e18afe6fc4245dd0600379624d9ef6fb0c, data reload: false

------ Round 1 ----------------------------------
q1	17637	4295	4230	4230
q2	2012	190	185	185
q3	10463	1156	1098	1098
q4	10187	759	829	759
q5	7560	2678	2648	2648
q6	217	135	133	133
q7	944	612	599	599
q8	9216	2053	2053	2053
q9	8672	6556	6551	6551
q10	8744	3765	3731	3731
q11	473	237	231	231
q12	451	223	214	214
q13	17842	3004	2967	2967
q14	275	233	249	233
q15	521	476	495	476
q16	497	385	370	370
q17	963	736	620	620
q18	7953	7463	7307	7307
q19	5623	1355	1391	1355
q20	672	337	332	332
q21	4864	3263	3216	3216
q22	339	284	289	284
Total cold run time: 116125 ms
Total hot run time: 39592 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4379	4278	4246	4246
q2	367	272	261	261
q3	2965	2757	2837	2757
q4	2019	1734	1721	1721
q5	5629	5526	5559	5526
q6	222	140	132	132
q7	2180	1820	1860	1820
q8	3283	3372	3354	3354
q9	8741	8699	8759	8699
q10	4139	3927	3830	3830
q11	591	489	485	485
q12	794	627	641	627
q13	17090	3156	3169	3156
q14	329	278	307	278
q15	525	481	486	481
q16	502	434	438	434
q17	1796	1534	1508	1508
q18	8086	8101	7886	7886
q19	3398	1563	1521	1521
q20	2229	1849	1862	1849
q21	5101	4943	4666	4666
q22	550	510	526	510
Total cold run time: 74915 ms
Total hot run time: 55747 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174134 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e15da6e18afe6fc4245dd0600379624d9ef6fb0c, data reload: false

query1	914	368	361	361
query2	6435	1844	1898	1844
query3	6641	205	213	205
query4	28391	17398	17465	17398
query5	3601	490	485	485
query6	269	191	178	178
query7	4603	300	285	285
query8	236	193	188	188
query9	8542	2426	2407	2407
query10	432	279	265	265
query11	10430	10078	9951	9951
query12	120	84	83	83
query13	1638	372	361	361
query14	9399	7809	7794	7794
query15	222	171	161	161
query16	7667	340	321	321
query17	1700	533	533	533
query18	1948	279	264	264
query19	203	145	142	142
query20	91	81	82	81
query21	202	130	136	130
query22	4325	4119	4048	4048
query23	33974	33740	33609	33609
query24	11079	3051	2909	2909
query25	617	435	426	426
query26	733	162	157	157
query27	2341	294	294	294
query28	6638	2113	2105	2105
query29	903	684	644	644
query30	257	158	158	158
query31	986	770	780	770
query32	96	55	56	55
query33	781	325	329	325
query34	927	504	502	502
query35	696	603	577	577
query36	1160	1046	1000	1000
query37	153	91	82	82
query38	3022	2858	2792	2792
query39	950	854	823	823
query40	212	124	125	124
query41	48	47	51	47
query42	118	101	103	101
query43	503	468	470	468
query44	1212	757	740	740
query45	191	167	167	167
query46	1087	759	719	719
query47	1850	1752	1786	1752
query48	376	309	295	295
query49	861	425	429	425
query50	779	404	391	391
query51	6997	6833	6800	6800
query52	119	90	96	90
query53	354	296	289	289
query54	910	464	466	464
query55	77	76	75	75
query56	307	291	305	291
query57	1123	1080	1037	1037
query58	253	286	262	262
query59	2739	2689	2578	2578
query60	333	304	305	304
query61	119	117	118	117
query62	790	663	645	645
query63	324	291	289	289
query64	9239	2381	1672	1672
query65	3181	3099	3110	3099
query66	758	339	352	339
query67	15889	15014	14965	14965
query68	6704	552	564	552
query69	775	422	364	364
query70	1218	1138	1150	1138
query71	543	280	282	280
query72	8693	5402	5441	5402
query73	812	321	320	320
query74	5969	5476	5488	5476
query75	4824	2688	2696	2688
query76	4728	959	915	915
query77	810	319	322	319
query78	10779	10125	9134	9134
query79	12747	516	522	516
query80	963	539	483	483
query81	569	226	222	222
query82	680	134	131	131
query83	317	167	164	164
query84	280	87	85	85
query85	1392	308	303	303
query86	426	276	323	276
query87	3321	3105	3150	3105
query88	5254	2465	2464	2464
query89	508	384	386	384
query90	2063	196	196	196
query91	130	105	100	100
query92	63	50	52	50
query93	6525	530	520	520
query94	1266	218	216	216
query95	460	313	321	313
query96	617	277	279	277
query97	3185	3001	3034	3001
query98	220	196	193	193
query99	1497	1289	1295	1289
Total cold run time: 302409 ms
Total hot run time: 174134 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e15da6e18afe6fc4245dd0600379624d9ef6fb0c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.06	0.06
query5	0.51	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.52	0.54	0.54
query11	0.17	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.75	0.77	0.82
query15	0.86	0.83	0.82
query16	0.37	0.37	0.37
query17	0.99	0.98	0.96
query18	0.23	0.22	0.21
query19	1.82	1.69	1.70
query20	0.01	0.00	0.01
query21	15.42	0.76	0.65
query22	4.43	6.61	2.00
query23	18.27	1.31	1.20
query24	2.11	0.23	0.23
query25	0.14	0.09	0.09
query26	0.29	0.22	0.20
query27	0.45	0.22	0.24
query28	13.29	1.02	1.00
query29	12.69	3.38	3.34
query30	0.25	0.06	0.06
query31	2.87	0.39	0.39
query32	3.26	0.46	0.48
query33	2.91	2.89	2.90
query34	17.18	4.36	4.37
query35	4.43	4.40	4.40
query36	0.65	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.16	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.11 s
Total hot run time: 30.66 s

dataroaring
dataroaring previously approved these changes Jul 13, 2024
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 13, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

need update

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 15, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40112 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a0c04caaecc9a2991bf583c5c67280e3ca321f1b, data reload: false

------ Round 1 ----------------------------------
q1	18795	4917	4398	4398
q2	2489	194	200	194
q3	11421	1169	1140	1140
q4	10690	796	759	759
q5	7611	2710	2707	2707
q6	225	139	138	138
q7	979	606	598	598
q8	9406	2035	2077	2035
q9	8611	6530	6531	6530
q10	8717	3726	3741	3726
q11	434	240	235	235
q12	384	227	220	220
q13	17890	2964	2957	2957
q14	269	239	232	232
q15	530	480	476	476
q16	507	382	372	372
q17	963	659	679	659
q18	8125	7535	7484	7484
q19	3948	1437	1409	1409
q20	675	328	325	325
q21	4869	3250	3234	3234
q22	353	295	284	284
Total cold run time: 117891 ms
Total hot run time: 40112 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4325	4290	4263	4263
q2	381	277	276	276
q3	2954	2799	2755	2755
q4	1874	1552	1601	1552
q5	5240	5295	5274	5274
q6	218	131	128	128
q7	2119	1763	1690	1690
q8	3179	3335	3303	3303
q9	8370	8341	8292	8292
q10	3859	3671	3733	3671
q11	576	494	490	490
q12	790	608	584	584
q13	17201	2929	2969	2929
q14	305	274	269	269
q15	530	487	486	486
q16	470	415	446	415
q17	1809	1476	1468	1468
q18	7656	7521	7373	7373
q19	3056	1473	1500	1473
q20	2022	1790	1763	1763
q21	4765	4675	4746	4675
q22	570	504	492	492
Total cold run time: 72269 ms
Total hot run time: 53621 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173445 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a0c04caaecc9a2991bf583c5c67280e3ca321f1b, data reload: false

query1	927	371	358	358
query2	6463	1901	1857	1857
query3	6667	202	218	202
query4	28072	17638	17569	17569
query5	4175	502	475	475
query6	255	167	175	167
query7	4601	302	286	286
query8	238	202	191	191
query9	8529	2354	2341	2341
query10	443	276	290	276
query11	11718	10359	10215	10215
query12	134	89	82	82
query13	1648	363	372	363
query14	10124	7043	7734	7043
query15	227	165	171	165
query16	7787	326	327	326
query17	1800	596	506	506
query18	1861	272	271	271
query19	203	148	146	146
query20	90	84	83	83
query21	205	130	126	126
query22	4619	4203	4264	4203
query23	33610	33241	32962	32962
query24	11935	2964	2856	2856
query25	655	369	383	369
query26	1778	145	142	142
query27	2908	264	273	264
query28	7273	1956	1959	1956
query29	1112	621	608	608
query30	287	147	145	145
query31	942	743	751	743
query32	90	53	53	53
query33	771	304	285	285
query34	982	481	497	481
query35	684	591	560	560
query36	1098	900	967	900
query37	289	82	80	80
query38	2866	2697	2767	2697
query39	860	814	811	811
query40	274	121	120	120
query41	47	46	44	44
query42	120	97	99	97
query43	509	463	472	463
query44	1198	740	733	733
query45	191	163	160	160
query46	1085	744	721	721
query47	1853	1782	1777	1777
query48	367	290	295	290
query49	1202	417	401	401
query50	782	397	399	397
query51	6955	6846	6889	6846
query52	100	92	93	92
query53	356	311	297	297
query54	997	440	439	439
query55	74	74	74	74
query56	287	259	274	259
query57	1188	1053	1093	1053
query58	241	240	258	240
query59	2870	2618	2603	2603
query60	311	269	269	269
query61	92	93	96	93
query62	820	645	666	645
query63	323	286	293	286
query64	10498	2210	7389	2210
query65	3156	3119	3120	3119
query66	1356	325	326	325
query67	15575	15259	15108	15108
query68	4478	535	526	526
query69	478	321	320	320
query70	1149	1143	1129	1129
query71	399	278	323	278
query72	6945	5871	6034	5871
query73	725	324	320	320
query74	6232	5619	5684	5619
query75	3371	2674	2705	2674
query76	2844	906	963	906
query77	466	317	303	303
query78	11045	9130	8913	8913
query79	2788	514	518	514
query80	1531	471	474	471
query81	576	218	227	218
query82	695	133	132	132
query83	270	174	169	169
query84	277	88	94	88
query85	1832	312	306	306
query86	492	308	311	308
query87	3303	3160	3118	3118
query88	4631	2455	2448	2448
query89	478	379	389	379
query90	1889	204	203	203
query91	140	110	114	110
query92	67	51	53	51
query93	4323	499	493	493
query94	1293	227	223	223
query95	420	399	309	309
query96	606	271	272	271
query97	3193	3005	3006	3005
query98	220	201	194	194
query99	1651	1252	1260	1252
Total cold run time: 290560 ms
Total hot run time: 173445 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a0c04caaecc9a2991bf583c5c67280e3ca321f1b, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.10	0.08
query5	0.51	0.49	0.50
query6	1.15	0.71	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.53	0.55	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.59	0.58
query14	0.77	0.77	0.78
query15	0.86	0.82	0.82
query16	0.36	0.35	0.36
query17	1.07	0.99	1.00
query18	0.22	0.21	0.22
query19	1.76	1.69	1.69
query20	0.01	0.01	0.01
query21	15.40	0.74	0.67
query22	3.93	7.86	1.90
query23	18.31	1.40	1.39
query24	2.13	0.23	0.22
query25	0.16	0.09	0.09
query26	0.29	0.21	0.21
query27	0.44	0.23	0.23
query28	13.23	1.01	1.02
query29	12.65	3.28	3.29
query30	0.26	0.06	0.06
query31	2.85	0.39	0.39
query32	3.27	0.48	0.47
query33	2.87	2.91	2.94
query34	17.09	4.31	4.34
query35	4.38	4.39	4.43
query36	0.64	0.49	0.48
query37	0.18	0.15	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.04
query40	0.14	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.34 s
Total hot run time: 30.71 s

Copy link
Collaborator

@yujun777 yujun777 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 17, 2024
@dataroaring dataroaring merged commit d7e172b into apache:master Jul 17, 2024
27 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
…xn does not block the other txns (#37724)

## Proposed changes

If any exception(such as NullPointerException) is thrown in
`genPublishTask` when publish, the publish for all txns will fail.
This pr catch the exception to make the failed txn does not block other
txns.
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jul 18, 2024
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jul 18, 2024
yiguolei pushed a commit that referenced this pull request Jul 18, 2024
dataroaring pushed a commit that referenced this pull request Jul 18, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants