Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[feature](Recycler) Add rate limit for all obj storage client implementation #37663

Merged
merged 9 commits into from
Jul 16, 2024
Merged

[feature](Recycler) Add rate limit for all obj storage client implementation #37663

merged 9 commits into from
Jul 16, 2024

Conversation

ByteYue
Copy link
Contributor

@ByteYue ByteYue commented Jul 11, 2024 •
edited by gavinchou
Loading

We should enable rate limit for all kinds of network IO of recycler. This pr adds the rate limiter for obj storage client.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 11, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

gavinchou
gavinchou reviewed Jul 13, 2024
View reviewed changes
Copy link
Contributor

@gavinchou gavinchou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BE may use the same rate limiter?

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 13, 2024

BE may use the same rate limiter?

Yes. I'll add it into BE later.

github-actions[bot]
github-actions bot reviewed Jul 14, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

cloud/test/s3_rate_limiter_test.cpp Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Jul 14, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

cloud/test/s3_rate_limiter_test.cpp Show resolved Hide resolved
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 14, 2024

run buildall

@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 14, 2024

run buildall

@ByteYue ByteYue requested a review from gavinchou July 14, 2024 16:07
@ByteYue
Copy link
Contributor Author

ByteYue commented Jul 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40327 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

------ Round 1 ----------------------------------
q1	17823	4528	4384	4384
q2	2545	201	195	195
q3	11792	1195	1190	1190
q4	10306	800	738	738
q5	7598	2750	2778	2750
q6	226	147	150	147
q7	971	621	612	612
q8	9480	2110	2110	2110
q9	8789	6606	6599	6599
q10	8835	3830	3756	3756
q11	448	238	239	238
q12	395	231	232	231
q13	17868	2993	2997	2993
q14	289	238	239	238
q15	530	495	488	488
q16	508	384	380	380
q17	995	646	663	646
q18	8247	7503	7403	7403
q19	6791	1422	1463	1422
q20	704	334	341	334
q21	4930	3181	3317	3181
q22	371	292	293	292
Total cold run time: 120441 ms
Total hot run time: 40327 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4346	4261	4285	4261
q2	375	276	266	266
q3	3092	2760	2769	2760
q4	1905	1567	1634	1567
q5	5304	5322	5349	5322
q6	220	134	134	134
q7	2169	1707	1738	1707
q8	3209	3395	3346	3346
q9	8503	8413	8435	8413
q10	3909	3727	3743	3727
q11	607	487	488	487
q12	775	624	606	606
q13	16903	3009	3006	3006
q14	299	271	275	271
q15	523	487	494	487
q16	491	422	435	422
q17	1773	1502	1456	1456
q18	7669	7512	7396	7396
q19	1712	1611	1449	1449
q20	2015	1828	1801	1801
q21	4846	4822	4647	4647
q22	575	495	494	494
Total cold run time: 71220 ms
Total hot run time: 54025 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173782 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

query1	910	365	367	365
query2	6452	1885	1864	1864
query3	6672	210	211	210
query4	28266	17649	17403	17403
query5	4180	521	490	490
query6	255	181	165	165
query7	4594	299	288	288
query8	242	197	183	183
query9	8590	2466	2427	2427
query10	446	309	274	274
query11	11715	10114	10041	10041
query12	130	85	80	80
query13	1626	362	375	362
query14	10388	7809	7819	7809
query15	220	171	167	167
query16	7605	314	315	314
query17	1742	536	533	533
query18	1255	274	275	274
query19	192	150	150	150
query20	91	84	83	83
query21	210	130	120	120
query22	4364	4102	3946	3946
query23	33776	33206	33176	33176
query24	11991	2841	2848	2841
query25	633	365	363	363
query26	1637	150	150	150
query27	2918	277	274	274
query28	7703	2044	2062	2044
query29	1026	627	607	607
query30	283	149	148	148
query31	951	743	760	743
query32	96	53	55	53
query33	780	296	290	290
query34	967	492	492	492
query35	695	609	568	568
query36	1130	944	944	944
query37	153	81	83	81
query38	2886	2720	2730	2720
query39	884	814	808	808
query40	279	132	127	127
query41	54	50	47	47
query42	122	96	107	96
query43	505	470	479	470
query44	1199	755	737	737
query45	194	167	163	163
query46	1092	763	736	736
query47	1857	1772	1783	1772
query48	375	293	298	293
query49	1211	445	431	431
query50	803	402	402	402
query51	6865	6860	6795	6795
query52	104	96	101	96
query53	357	297	311	297
query54	1050	462	474	462
query55	80	75	78	75
query56	305	296	288	288
query57	1161	1059	1078	1059
query58	274	249	267	249
query59	3037	2600	2564	2564
query60	333	299	304	299
query61	116	114	113	113
query62	825	664	659	659
query63	334	301	293	293
query64	10665	2308	1807	1807
query65	3318	3120	3113	3113
query66	1286	346	368	346
query67	15757	15128	15020	15020
query68	4647	566	562	562
query69	624	466	358	358
query70	1232	1156	1052	1052
query71	436	296	289	289
query72	8578	6208	5647	5647
query73	758	342	327	327
query74	6156	5640	5727	5640
query75	3719	2741	2674	2674
query76	3036	921	960	921
query77	674	328	321	321
query78	9623	9113	9053	9053
query79	9109	537	536	536
query80	1858	520	495	495
query81	583	225	222	222
query82	1631	139	134	134
query83	358	169	174	169
query84	275	83	89	83
query85	2049	308	298	298
query86	446	334	320	320
query87	3332	3078	3114	3078
query88	5568	2393	2393	2393
query89	527	405	386	386
query90	1813	198	197	197
query91	133	99	104	99
query92	63	51	52	51
query93	6715	527	519	519
query94	1097	219	225	219
query95	414	322	333	322
query96	620	273	279	273
query97	3194	3086	3041	3041
query98	232	204	196	196
query99	1562	1248	1267	1248
Total cold run time: 303274 ms
Total hot run time: 173782 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba800d8fbfa05a4ccbe500fcc9438e587ab9a4a9, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.49	0.50
query6	1.14	0.74	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.54	0.52	0.55
query11	0.15	0.11	0.11
query12	0.15	0.13	0.13
query13	0.60	0.59	0.59
query14	0.79	0.77	0.79
query15	0.86	0.82	0.81
query16	0.34	0.35	0.37
query17	0.98	0.98	1.01
query18	0.24	0.23	0.23
query19	1.86	1.74	1.72
query20	0.02	0.01	0.01
query21	15.41	0.73	0.65
query22	4.00	7.53	1.88
query23	18.25	1.26	1.21
query24	2.10	0.23	0.23
query25	0.15	0.09	0.09
query26	0.28	0.20	0.20
query27	0.46	0.23	0.23
query28	13.20	1.02	1.01
query29	12.58	3.36	3.35
query30	0.25	0.06	0.05
query31	2.87	0.39	0.38
query32	3.28	0.49	0.47
query33	2.90	2.85	2.94
query34	16.88	4.35	4.37
query35	4.43	4.44	4.40
query36	0.66	0.46	0.50
query37	0.20	0.15	0.15
query38	0.15	0.15	0.14
query39	0.05	0.04	0.03
query40	0.15	0.12	0.11
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.25 s
Total hot run time: 30.56 s

@ByteYue ByteYue requested a review from gavinchou July 15, 2024 09:12
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 0ad6289 into apache:master Jul 16, 2024
26 of 30 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@ByteYue @seawinde
[feature](Recycler) Add rate limit for all obj storage client impleme…
14dc0fe 
…ntation (apache#37663)

We should enable rate limit for all kinds of network IO of recycler.
This pr adds the rate limiter for obj storage client.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@ByteYue @dataroaring
[feature](Recycler) Add rate limit for all obj storage client impleme…
48c537c 
…ntation (#37663)

We should enable rate limit for all kinds of network IO of recycler.
This pr adds the rate limiter for obj storage client.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@platoneko platoneko platoneko approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@ByteYue @doris-robot @gavinchou @platoneko @dataroaring