Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve](routine-load) add retry when get Kafka meta info #35376

Merged
merged 1 commit into from
Jun 5, 2024
Merged

[improve](routine-load) add retry when get Kafka meta info #35376

merged 1 commit into from
Jun 5, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented May 24, 2024

Proposed changes

If be down when FE send RPC getInfo or meet network error when be send RPC to Kafka, routine load job will pause.
To keep routine load stable, add retry when get Kafka meta info.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui sollhui force-pushed the get_kafka_meta_add_retry branch from 0aa90fb to b028e30 Compare May 24, 2024 11:39
@sollhui
Copy link
Contributor Author

sollhui commented May 24, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41813 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b028e309a4a3337740a18ffc19b7049fa8eaa37c, data reload: false

------ Round 1 ----------------------------------
q1	17619	4477	4221	4221
q2	2012	182	189	182
q3	10475	1219	1206	1206
q4	10205	827	822	822
q5	7468	2719	2722	2719
q6	223	130	131	130
q7	960	610	589	589
q8	9218	2093	2100	2093
q9	9810	6653	6722	6653
q10	9003	3867	3904	3867
q11	439	239	245	239
q12	459	234	238	234
q13	17215	3253	3225	3225
q14	270	230	238	230
q15	520	488	473	473
q16	513	407	406	406
q17	997	732	629	629
q18	8281	7774	7924	7774
q19	8099	1559	1558	1558
q20	659	322	351	322
q21	5111	3970	4338	3970
q22	355	272	271	271
Total cold run time: 119911 ms
Total hot run time: 41813 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4495	4466	4432	4432
q2	361	270	271	270
q3	3142	2856	2893	2856
q4	1967	1679	1627	1627
q5	5308	5464	5482	5464
q6	210	121	118	118
q7	2233	1822	1811	1811
q8	3244	3418	3406	3406
q9	8616	8604	8637	8604
q10	4117	3772	3697	3697
q11	580	475	488	475
q12	780	608	620	608
q13	15926	3128	3155	3128
q14	295	264	273	264
q15	526	479	479	479
q16	482	442	462	442
q17	1819	1541	1517	1517
q18	7828	7558	7391	7391
q19	1665	1566	1495	1495
q20	2031	1794	1787	1787
q21	10440	4792	4730	4730
q22	581	496	477	477
Total cold run time: 76646 ms
Total hot run time: 55078 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171721 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b028e309a4a3337740a18ffc19b7049fa8eaa37c, data reload: false

query1	902	378	376	376
query2	6458	2349	2268	2268
query3	6647	201	210	201
query4	19210	17336	17150	17150
query5	4097	416	419	416
query6	242	153	154	153
query7	4590	291	295	291
query8	232	184	177	177
query9	8662	2406	2396	2396
query10	448	275	260	260
query11	10502	10121	10150	10121
query12	133	92	87	87
query13	1643	371	376	371
query14	9240	7580	6749	6749
query15	243	166	171	166
query16	8008	264	258	258
query17	1709	519	525	519
query18	2048	267	266	266
query19	211	152	155	152
query20	93	86	90	86
query21	202	127	130	127
query22	4351	4259	3999	3999
query23	33806	33165	33063	33063
query24	9936	2855	2821	2821
query25	555	350	368	350
query26	714	149	153	149
query27	2214	316	322	316
query28	5570	2092	2084	2084
query29	837	595	583	583
query30	230	178	170	170
query31	967	743	717	717
query32	98	51	52	51
query33	648	262	254	254
query34	877	467	480	467
query35	681	604	598	598
query36	1089	921	881	881
query37	105	69	68	68
query38	2901	2747	2787	2747
query39	827	783	783	783
query40	201	126	123	123
query41	44	43	43	43
query42	106	97	98	97
query43	575	541	552	541
query44	1086	742	752	742
query45	173	160	159	159
query46	1077	720	714	714
query47	1866	1740	1768	1740
query48	363	300	297	297
query49	843	372	389	372
query50	759	397	389	389
query51	6845	6719	6708	6708
query52	97	87	92	87
query53	347	278	279	278
query54	849	429	435	429
query55	74	72	73	72
query56	285	234	233	233
query57	1124	1065	1032	1032
query58	228	208	205	205
query59	3398	3022	3006	3006
query60	266	259	266	259
query61	87	86	86	86
query62	593	455	445	445
query63	310	287	279	279
query64	8540	2200	1713	1713
query65	3143	3060	3114	3060
query66	792	336	327	327
query67	15205	15101	14918	14918
query68	4547	543	544	543
query69	495	260	267	260
query70	1149	1109	1129	1109
query71	393	278	311	278
query72	7020	5424	5390	5390
query73	732	329	319	319
query74	5931	5544	5610	5544
query75	3454	2633	2622	2622
query76	2795	1068	1015	1015
query77	593	268	259	259
query78	10295	9907	9808	9808
query79	1399	517	524	517
query80	845	441	421	421
query81	540	249	237	237
query82	671	91	97	91
query83	239	168	165	165
query84	246	88	82	82
query85	1090	329	259	259
query86	413	322	306	306
query87	3296	3124	3147	3124
query88	3293	2340	2337	2337
query89	469	387	367	367
query90	1929	185	184	184
query91	121	96	95	95
query92	61	47	47	47
query93	1081	522	505	505
query94	1225	181	185	181
query95	393	308	304	304
query96	583	266	265	265
query97	3201	2988	3044	2988
query98	225	218	216	216
query99	1102	861	838	838
Total cold run time: 261028 ms
Total hot run time: 171721 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b028e309a4a3337740a18ffc19b7049fa8eaa37c, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.06
query5	0.51	0.49	0.51
query6	1.12	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.53	0.46	0.49
query10	0.55	0.56	0.53
query11	0.15	0.12	0.11
query12	0.14	0.11	0.12
query13	0.59	0.58	0.59
query14	0.77	0.76	0.76
query15	0.82	0.81	0.81
query16	0.36	0.35	0.37
query17	1.00	0.94	1.03
query18	0.21	0.25	0.24
query19	1.72	1.64	1.71
query20	0.01	0.02	0.01
query21	15.73	0.66	0.65
query22	3.83	8.39	1.63
query23	18.28	1.35	1.31
query24	1.91	0.23	0.21
query25	0.13	0.09	0.09
query26	0.28	0.16	0.17
query27	0.08	0.08	0.08
query28	13.38	1.01	1.00
query29	13.50	3.36	3.27
query30	0.25	0.06	0.05
query31	3.20	0.40	0.39
query32	3.70	0.47	0.47
query33	2.94	3.04	2.91
query34	17.15	4.42	4.42
query35	4.54	4.53	4.60
query36	0.66	0.46	0.46
query37	0.17	0.14	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.15	0.16
query41	0.09	0.04	0.04
query42	0.05	0.05	0.05
query43	0.04	0.03	0.05
Total cold run time: 110.83 s
Total hot run time: 30.19 s

@sollhui sollhui force-pushed the get_kafka_meta_add_retry branch from ab925f9 to 860d1bd Compare May 30, 2024 10:02
@sollhui
Copy link
Contributor Author

sollhui commented May 30, 2024

run buildall

@sollhui sollhui force-pushed the get_kafka_meta_add_retry branch 2 times, most recently from 026b3a3 to 763df1a Compare May 30, 2024 12:36
@sollhui
Copy link
Contributor Author

sollhui commented May 30, 2024

run buildall

@sollhui sollhui force-pushed the get_kafka_meta_add_retry branch 2 times, most recently from efd4041 to c44453b Compare May 30, 2024 12:39
@sollhui
Copy link
Contributor Author

sollhui commented May 30, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41429 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c44453be3de5a60ecdfb1305a4272c7d23197637, data reload: false

------ Round 1 ----------------------------------
q1	17623	4378	4263	4263
q2	2031	203	197	197
q3	10624	1295	1301	1295
q4	10263	766	876	766
q5	7617	2699	2713	2699
q6	225	139	144	139
q7	992	610	607	607
q8	9313	2107	2135	2107
q9	9481	6702	6686	6686
q10	9104	3747	3717	3717
q11	455	249	246	246
q12	476	228	234	228
q13	17785	3003	3007	3003
q14	265	234	227	227
q15	523	485	484	484
q16	493	385	387	385
q17	974	742	667	667
q18	8033	7539	7511	7511
q19	3055	1632	1578	1578
q20	658	315	307	307
q21	5023	3991	4039	3991
q22	384	335	326	326
Total cold run time: 115397 ms
Total hot run time: 41429 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4365	4278	4263	4263
q2	387	269	268	268
q3	3003	2800	2684	2684
q4	1923	1609	1643	1609
q5	5278	5275	5274	5274
q6	211	128	133	128
q7	2100	1779	1785	1779
q8	3224	3361	3318	3318
q9	8419	8372	8420	8372
q10	3949	3730	3744	3730
q11	591	502	488	488
q12	765	617	608	608
q13	16328	3003	3006	3003
q14	302	261	264	261
q15	518	474	483	474
q16	469	410	410	410
q17	1800	1504	1478	1478
q18	7686	7531	7393	7393
q19	1721	1556	1591	1556
q20	2009	1794	1787	1787
q21	4875	4646	4754	4646
q22	609	523	532	523
Total cold run time: 70532 ms
Total hot run time: 54052 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170644 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c44453be3de5a60ecdfb1305a4272c7d23197637, data reload: false

query1	903	381	367	367
query2	6453	2456	2341	2341
query3	6656	217	218	217
query4	19442	17476	17323	17323
query5	4187	427	431	427
query6	249	160	154	154
query7	4586	294	294	294
query8	338	284	295	284
query9	8598	2394	2384	2384
query10	449	298	260	260
query11	10730	10149	10124	10124
query12	140	91	90	90
query13	1673	357	360	357
query14	10123	7664	7636	7636
query15	224	189	192	189
query16	8028	261	261	261
query17	1848	556	532	532
query18	1991	273	272	272
query19	205	159	156	156
query20	94	87	88	87
query21	197	132	133	132
query22	4383	3949	3936	3936
query23	33616	33173	32940	32940
query24	12064	2920	2924	2920
query25	651	342	356	342
query26	1796	156	158	156
query27	3086	316	328	316
query28	7698	2066	2077	2066
query29	1106	630	600	600
query30	304	149	154	149
query31	964	763	761	761
query32	91	52	55	52
query33	777	270	258	258
query34	1037	477	486	477
query35	726	613	595	595
query36	1087	937	910	910
query37	267	66	71	66
query38	2904	2817	2723	2723
query39	841	799	795	795
query40	274	128	127	127
query41	54	52	52	52
query42	107	97	98	97
query43	605	573	553	553
query44	1254	727	739	727
query45	182	173	169	169
query46	1052	706	697	697
query47	1840	1750	1775	1750
query48	386	309	297	297
query49	1226	384	389	384
query50	763	397	390	390
query51	6923	6694	6739	6694
query52	98	99	90	90
query53	356	291	285	285
query54	906	431	431	431
query55	77	72	75	72
query56	285	252	244	244
query57	1126	1069	1032	1032
query58	244	210	200	200
query59	3754	3441	3489	3441
query60	290	254	257	254
query61	89	92	91	91
query62	670	444	442	442
query63	314	289	296	289
query64	9935	2204	1776	1776
query65	3178	3105	3110	3105
query66	1396	329	333	329
query67	15201	14762	15008	14762
query68	5301	541	530	530
query69	484	272	272	272
query70	1138	1116	1133	1116
query71	450	272	268	268
query72	7755	2676	2510	2510
query73	766	326	319	319
query74	6031	5657	5637	5637
query75	3772	2619	2657	2619
query76	3748	1046	1002	1002
query77	638	272	268	268
query78	10188	9953	9852	9852
query79	1305	522	519	519
query80	709	445	433	433
query81	494	225	218	218
query82	756	95	96	95
query83	193	169	169	169
query84	237	88	85	85
query85	1406	282	370	282
query86	416	294	299	294
query87	3305	3138	3077	3077
query88	3317	2327	2338	2327
query89	485	402	385	385
query90	1978	190	192	190
query91	127	99	100	99
query92	60	50	54	50
query93	1221	508	498	498
query94	1136	196	185	185
query95	401	318	309	309
query96	577	259	262	259
query97	3215	2987	3020	2987
query98	236	226	222	222
query99	1181	839	843	839
Total cold run time: 276123 ms
Total hot run time: 170644 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.29 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c44453be3de5a60ecdfb1305a4272c7d23197637, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.08	0.08
query5	0.51	0.48	0.50
query6	1.12	0.72	0.73
query7	0.01	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.49
query10	0.55	0.55	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.78	0.81
query15	0.83	0.80	0.82
query16	0.37	0.37	0.36
query17	0.96	0.96	0.98
query18	0.23	0.22	0.24
query19	1.76	1.67	1.77
query20	0.02	0.01	0.01
query21	15.54	0.69	0.64
query22	4.15	7.26	1.78
query23	18.37	1.46	1.24
query24	1.91	0.25	0.21
query25	0.14	0.08	0.08
query26	0.28	0.16	0.16
query27	0.08	0.09	0.09
query28	13.32	1.02	0.99
query29	13.18	3.33	3.37
query30	0.24	0.06	0.06
query31	2.91	0.40	0.38
query32	3.25	0.48	0.47
query33	2.91	2.88	2.90
query34	17.17	4.37	4.36
query35	4.48	4.50	4.47
query36	0.65	0.46	0.47
query37	0.18	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.15
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.98 s
Total hot run time: 30.29 s

@sollhui sollhui force-pushed the get_kafka_meta_add_retry branch from c44453b to cf96304 Compare May 31, 2024 03:38
@sollhui
Copy link
Contributor Author

sollhui commented May 31, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40026 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf963043a71214016100ad3b8551848eab0da8cc, data reload: false

------ Round 1 ----------------------------------
q1	18179	4423	4481	4423
q2	2659	201	191	191
q3	12113	1151	1167	1151
q4	10428	697	780	697
q5	7523	2722	2653	2653
q6	224	133	135	133
q7	975	621	602	602
q8	9365	2045	2095	2045
q9	9016	6496	6497	6496
q10	8860	3713	3738	3713
q11	457	249	237	237
q12	545	221	227	221
q13	17759	2986	2980	2980
q14	269	225	213	213
q15	504	464	456	456
q16	536	375	385	375
q17	964	744	721	721
q18	8149	7398	7321	7321
q19	5704	1593	1573	1573
q20	659	304	304	304
q21	4912	3197	3226	3197
q22	374	324	333	324
Total cold run time: 120174 ms
Total hot run time: 40026 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4235	4206	4206
q2	379	271	274	271
q3	2940	2747	2726	2726
q4	1859	1604	1611	1604
q5	5235	5274	5273	5273
q6	211	126	125	125
q7	2085	1754	1727	1727
q8	3213	3288	3316	3288
q9	8314	8295	8303	8295
q10	3862	3689	3673	3673
q11	580	479	490	479
q12	767	599	589	589
q13	16323	2998	3000	2998
q14	293	263	264	263
q15	516	487	485	485
q16	466	408	413	408
q17	1763	1479	1473	1473
q18	7696	7605	7441	7441
q19	2030	1602	1559	1559
q20	2005	1772	1756	1756
q21	6275	4710	4648	4648
q22	605	514	539	514
Total cold run time: 71769 ms
Total hot run time: 53801 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169007 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf963043a71214016100ad3b8551848eab0da8cc, data reload: false

query1	941	372	375	372
query2	6459	2548	2439	2439
query3	6663	205	199	199
query4	19237	17377	17346	17346
query5	4144	418	420	418
query6	251	157	155	155
query7	4597	294	286	286
query8	303	285	292	285
query9	8488	2372	2360	2360
query10	458	295	271	271
query11	10596	10138	10111	10111
query12	135	91	88	88
query13	1637	360	355	355
query14	9351	6778	7519	6778
query15	236	190	191	190
query16	7825	280	255	255
query17	1843	522	532	522
query18	1967	279	268	268
query19	204	165	151	151
query20	93	88	86	86
query21	200	133	128	128
query22	4101	3994	3879	3879
query23	33577	33095	32949	32949
query24	11999	2841	2787	2787
query25	659	361	379	361
query26	1802	156	153	153
query27	2993	308	311	308
query28	7300	2042	2032	2032
query29	1136	614	593	593
query30	294	151	153	151
query31	977	752	748	748
query32	97	52	54	52
query33	752	260	279	260
query34	998	484	481	481
query35	754	627	632	627
query36	1088	947	916	916
query37	263	70	69	69
query38	2919	2786	2771	2771
query39	870	795	785	785
query40	282	126	127	126
query41	55	53	55	53
query42	107	100	97	97
query43	581	592	570	570
query44	1229	731	738	731
query45	194	171	172	171
query46	1070	737	701	701
query47	1916	1761	1804	1761
query48	362	299	288	288
query49	1214	392	383	383
query50	771	383	397	383
query51	6837	6682	6577	6577
query52	107	89	93	89
query53	349	291	289	289
query54	899	448	417	417
query55	75	71	71	71
query56	258	238	245	238
query57	1123	1055	1061	1055
query58	226	206	218	206
query59	3741	3075	3227	3075
query60	275	252	258	252
query61	96	89	90	89
query62	649	442	452	442
query63	316	282	286	282
query64	9868	2253	1716	1716
query65	3204	3097	3153	3097
query66	1385	329	342	329
query67	15231	14736	14818	14736
query68	4609	534	533	533
query69	434	265	267	265
query70	1157	1095	1138	1095
query71	403	272	268	268
query72	7672	2724	2551	2551
query73	712	317	320	317
query74	6093	5619	5769	5619
query75	3323	2656	2626	2626
query76	2619	975	955	955
query77	479	267	259	259
query78	10294	9785	9737	9737
query79	2142	507	502	502
query80	921	445	427	427
query81	511	219	219	219
query82	707	96	88	88
query83	236	172	176	172
query84	233	85	86	85
query85	1960	301	274	274
query86	478	313	308	308
query87	3345	3119	3114	3114
query88	4189	2330	2353	2330
query89	474	396	386	386
query90	2002	185	181	181
query91	129	98	97	97
query92	60	47	48	47
query93	2409	510	489	489
query94	1247	199	193	193
query95	412	320	323	320
query96	591	265	267	265
query97	3225	3034	2977	2977
query98	241	225	227	225
query99	1115	874	889	874
Total cold run time: 275049 ms
Total hot run time: 169007 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf963043a71214016100ad3b8551848eab0da8cc, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.09	0.09
query5	0.51	0.48	0.51
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.53	0.50	0.50
query10	0.54	0.54	0.55
query11	0.15	0.11	0.11
query12	0.16	0.12	0.12
query13	0.59	0.60	0.61
query14	0.79	0.77	0.79
query15	0.83	0.80	0.81
query16	0.35	0.35	0.36
query17	1.02	0.97	1.00
query18	0.21	0.25	0.24
query19	1.76	1.66	1.66
query20	0.01	0.02	0.01
query21	15.70	0.68	0.67
query22	4.16	7.64	1.55
query23	18.27	1.44	1.34
query24	1.34	0.39	0.20
query25	0.15	0.09	0.08
query26	0.26	0.16	0.16
query27	0.07	0.08	0.08
query28	13.50	1.03	1.00
query29	13.19	3.24	3.25
query30	0.25	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.47	0.46
query33	2.85	2.90	2.85
query34	17.27	4.43	4.44
query35	4.48	4.49	4.57
query36	0.65	0.47	0.46
query37	0.17	0.16	0.16
query38	0.16	0.14	0.14
query39	0.05	0.04	0.04
query40	0.17	0.15	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.7 s
Total hot run time: 30.17 s

liaoxin01
liaoxin01 approved these changes Jun 4, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 4, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring approved these changes Jun 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring force-pushed the get_kafka_meta_add_retry branch from cf96304 to dca50ae Compare June 5, 2024 01:11
@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40805 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit dca50aec42587d79f4e94453cbbcf56a6f810592, data reload: false

------ Round 1 ----------------------------------
q1	17629	4333	4251	4251
q2	2026	217	189	189
q3	10445	1206	1110	1110
q4	10186	871	855	855
q5	7492	2687	2664	2664
q6	217	133	135	133
q7	965	599	607	599
q8	9217	2116	2078	2078
q9	9366	6745	6704	6704
q10	9300	3958	3889	3889
q11	459	240	242	240
q12	432	224	246	224
q13	17252	3139	3242	3139
q14	274	227	236	227
q15	509	466	485	466
q16	503	367	381	367
q17	961	603	623	603
q18	8322	7741	7683	7683
q19	7847	1558	1529	1529
q20	648	326	324	324
q21	5060	3188	3903	3188
q22	408	350	343	343
Total cold run time: 119518 ms
Total hot run time: 40805 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4802	4455	4483	4455
q2	370	265	271	265
q3	3185	2977	2976	2976
q4	1907	1666	1623	1623
q5	5409	5516	5449	5449
q6	227	125	123	123
q7	2238	1850	1851	1850
q8	3223	3389	3384	3384
q9	8638	8639	8676	8639
q10	4034	3838	3723	3723
q11	575	503	481	481
q12	816	627	639	627
q13	16090	3113	3084	3084
q14	304	271	276	271
q15	507	473	487	473
q16	473	434	428	428
q17	1826	1508	1464	1464
q18	7997	7515	7272	7272
q19	1768	1506	1539	1506
q20	3022	1815	1806	1806
q21	9144	4801	4775	4775
q22	707	548	542	542
Total cold run time: 77262 ms
Total hot run time: 55216 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173106 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit dca50aec42587d79f4e94453cbbcf56a6f810592, data reload: false

query1	923	374	371	371
query2	6477	2494	2531	2494
query3	7216	212	215	212
query4	18998	17262	17175	17175
query5	4137	462	483	462
query6	241	163	154	154
query7	4595	305	303	303
query8	329	291	277	277
query9	8491	2457	2426	2426
query10	435	299	296	296
query11	10588	10029	9905	9905
query12	137	88	87	87
query13	1634	367	386	367
query14	10152	7741	6894	6894
query15	236	184	183	183
query16	7742	276	268	268
query17	1515	530	537	530
query18	1926	273	270	270
query19	198	150	161	150
query20	93	88	91	88
query21	204	132	139	132
query22	4436	4170	3967	3967
query23	33711	33191	33182	33182
query24	10976	2786	2836	2786
query25	613	373	368	368
query26	1195	158	168	158
query27	2978	332	332	332
query28	7247	2090	2118	2090
query29	923	613	600	600
query30	284	147	151	147
query31	986	746	741	741
query32	96	52	56	52
query33	761	279	281	279
query34	944	486	478	478
query35	733	645	604	604
query36	1104	944	914	914
query37	148	69	71	69
query38	2851	2721	2744	2721
query39	851	797	817	797
query40	215	130	129	129
query41	58	54	56	54
query42	114	102	100	100
query43	590	571	583	571
query44	1223	731	743	731
query45	208	181	170	170
query46	1085	730	722	722
query47	1842	1783	1751	1751
query48	366	310	306	306
query49	1006	410	461	410
query50	789	387	387	387
query51	6904	6852	6784	6784
query52	106	99	92	92
query53	352	298	295	295
query54	844	462	450	450
query55	75	76	77	76
query56	284	259	267	259
query57	1121	1042	1048	1042
query58	273	240	253	240
query59	3512	3426	3348	3348
query60	289	277	299	277
query61	92	120	88	88
query62	622	444	428	428
query63	323	294	290	290
query64	8935	2239	1796	1796
query65	3156	3099	3148	3099
query66	879	366	327	327
query67	15323	14916	15161	14916
query68	5584	556	543	543
query69	633	461	396	396
query70	1180	1036	1102	1036
query71	449	288	284	284
query72	7389	5599	5368	5368
query73	789	338	334	334
query74	5854	5455	5529	5455
query75	3751	2683	2646	2646
query76	3303	960	871	871
query77	630	300	300	300
query78	10280	9674	9739	9674
query79	1604	514	515	514
query80	852	474	467	467
query81	571	228	227	227
query82	741	109	102	102
query83	222	178	177	177
query84	257	89	88	88
query85	1371	311	337	311
query86	450	309	317	309
query87	3264	3063	3074	3063
query88	3631	2450	2448	2448
query89	467	402	377	377
query90	1819	189	193	189
query91	126	99	99	99
query92	64	53	51	51
query93	1745	525	499	499
query94	1117	193	191	191
query95	418	318	334	318
query96	585	272	266	266
query97	3215	2976	3044	2976
query98	252	220	220	220
query99	1216	840	885	840
Total cold run time: 271519 ms
Total hot run time: 173106 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit dca50aec42587d79f4e94453cbbcf56a6f810592, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.05
query3	0.24	0.05	0.04
query4	1.68	0.08	0.08
query5	0.51	0.46	0.49
query6	1.13	0.72	0.72
query7	0.01	0.01	0.01
query8	0.05	0.04	0.04
query9	0.53	0.49	0.49
query10	0.54	0.56	0.53
query11	0.15	0.11	0.12
query12	0.16	0.12	0.12
query13	0.60	0.58	0.60
query14	0.78	0.77	0.78
query15	0.83	0.81	0.82
query16	0.36	0.34	0.35
query17	1.02	0.99	0.98
query18	0.21	0.23	0.26
query19	1.83	1.71	1.70
query20	0.02	0.01	0.01
query21	15.48	0.69	0.66
query22	5.03	5.73	2.47
query23	18.30	1.37	1.14
query24	1.52	0.40	0.20
query25	0.14	0.07	0.08
query26	0.28	0.17	0.18
query27	0.07	0.07	0.08
query28	13.33	1.01	0.98
query29	13.34	3.38	3.32
query30	0.23	0.05	0.06
query31	2.89	0.39	0.38
query32	3.28	0.47	0.47
query33	2.90	2.88	2.93
query34	17.32	4.44	4.42
query35	4.46	4.49	4.55
query36	0.70	0.47	0.46
query37	0.18	0.15	0.14
query38	0.15	0.14	0.15
query39	0.04	0.04	0.03
query40	0.17	0.13	0.14
query41	0.08	0.05	0.04
query42	0.06	0.05	0.04
query43	0.03	0.04	0.04
Total cold run time: 110.75 s
Total hot run time: 30.88 s

dataroaring
dataroaring approved these changes Jun 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit f732052 into apache:master Jun 5, 2024
26 of 27 checks passed
dataroaring pushed a commit that referenced this pull request Jun 7, 2024
@sollhui @dataroaring
[improve](routine-load) add retry when get Kafka meta info (#35376)
a5321eb 
If be down when FE send RPC `getInfo` or meet network error when be send
RPC to Kafka, routine load job will pause.
To keep routine load stable, add retry when get Kafka meta info.
@sollhui sollhui mentioned this pull request Jul 6, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 7, 2024
@sollhui
[branch-2.1](routine-load) add retry when get Kafka meta info (#37371)
960b02f 
pick #35376
@sollhui sollhui mentioned this pull request Jul 8, 2024
Merged
@xiaokang xiaokang mentioned this pull request Jul 14, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @dataroaring @liaoxin01