Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](compaction) optimizing memory usage for compaction #37099

Merged
merged 11 commits into from
Jul 2, 2024
Merged

[enhancement](compaction) optimizing memory usage for compaction #37099

merged 11 commits into from
Jul 2, 2024

Conversation

luwei16
Copy link
Contributor

@luwei16 luwei16 commented Jul 1, 2024

No description provided.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actions bot reviewed Jul 1, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/test/olap/base_compaction_test.cpp
// specific language governing permissions and limitations
// under the License.

#include "olap/base_compaction.h"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'olap/base_compaction.h' file not found [clang-diagnostic-error]

#include "olap/base_compaction.h"
         ^

@luwei16
Copy link
Contributor Author

luwei16 commented Jul 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40050 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9870a88e0cc07a984b98ed5b3ae62fe05e5e7333, data reload: false

------ Round 1 ----------------------------------
q1	17593	4435	4341	4341
q2	2009	186	189	186
q3	10496	1203	1095	1095
q4	10197	800	730	730
q5	7476	2703	2657	2657
q6	229	141	139	139
q7	962	592	603	592
q8	9224	2127	2095	2095
q9	8896	6537	6465	6465
q10	9019	3745	3711	3711
q11	441	250	240	240
q12	499	242	230	230
q13	18695	2950	2962	2950
q14	279	231	215	215
q15	527	481	501	481
q16	516	375	378	375
q17	983	679	690	679
q18	8046	7466	7404	7404
q19	4579	1547	1443	1443
q20	666	331	334	331
q21	4939	3873	3357	3357
q22	407	336	334	334
Total cold run time: 116678 ms
Total hot run time: 40050 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4530	4265	4296	4265
q2	370	270	269	269
q3	2959	2889	2868	2868
q4	2032	1691	1706	1691
q5	5656	5510	5456	5456
q6	236	133	138	133
q7	2205	1869	1824	1824
q8	3279	3419	3473	3419
q9	8749	8745	8775	8745
q10	4163	3809	3849	3809
q11	604	522	513	513
q12	815	654	648	648
q13	15979	3139	3189	3139
q14	305	295	285	285
q15	508	498	491	491
q16	506	441	452	441
q17	1850	1537	1528	1528
q18	8300	7874	7806	7806
q19	1799	1620	1703	1620
q20	2193	1867	1876	1867
q21	9040	4834	4779	4779
q22	648	550	559	550
Total cold run time: 76726 ms
Total hot run time: 56146 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172862 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9870a88e0cc07a984b98ed5b3ae62fe05e5e7333, data reload: false

query1	917	383	383	383
query2	6447	2628	2443	2443
query3	6638	210	219	210
query4	19661	17429	17202	17202
query5	3753	488	491	488
query6	255	163	169	163
query7	4585	307	303	303
query8	310	278	286	278
query9	8582	2405	2374	2374
query10	578	303	283	283
query11	10522	10000	9969	9969
query12	120	84	80	80
query13	1664	372	368	368
query14	10128	6973	6235	6235
query15	223	181	193	181
query16	7775	281	276	276
query17	1700	579	558	558
query18	1979	291	295	291
query19	202	160	165	160
query20	90	81	85	81
query21	221	132	129	129
query22	4202	4129	3922	3922
query23	33791	33347	33504	33347
query24	10906	2926	2968	2926
query25	622	416	409	409
query26	714	168	165	165
query27	2255	329	327	327
query28	5841	2180	2165	2165
query29	931	672	655	655
query30	277	161	160	160
query31	960	777	750	750
query32	102	56	64	56
query33	670	307	301	301
query34	891	488	500	488
query35	781	669	651	651
query36	1138	992	1002	992
query37	148	83	83	83
query38	2970	2794	2832	2794
query39	887	830	832	830
query40	217	146	134	134
query41	57	54	52	52
query42	110	108	103	103
query43	624	530	549	530
query44	1103	724	734	724
query45	205	167	166	166
query46	1082	736	752	736
query47	1813	1812	1767	1767
query48	390	302	307	302
query49	858	517	425	425
query50	786	392	391	391
query51	6842	6757	6696	6696
query52	99	99	94	94
query53	369	298	301	298
query54	882	453	435	435
query55	77	76	87	76
query56	293	263	268	263
query57	1125	1064	1018	1018
query58	243	254	242	242
query59	3586	3396	3129	3129
query60	313	285	317	285
query61	96	89	95	89
query62	617	463	453	453
query63	316	299	293	293
query64	8830	2317	1784	1784
query65	3185	3092	3122	3092
query66	738	329	333	329
query67	15562	14860	14876	14860
query68	7216	534	550	534
query69	693	420	385	385
query70	1190	1172	1091	1091
query71	511	284	276	276
query72	8789	5376	5519	5376
query73	804	325	327	325
query74	5848	5447	5378	5378
query75	5273	2705	2662	2662
query76	4562	921	971	921
query77	762	314	304	304
query78	10371	9806	9761	9761
query79	7644	521	525	521
query80	1044	482	493	482
query81	546	227	223	223
query82	777	111	110	110
query83	312	176	174	174
query84	268	85	86	85
query85	1263	290	274	274
query86	456	301	309	301
query87	3276	3074	3097	3074
query88	4565	2372	2363	2363
query89	537	394	404	394
query90	1877	193	190	190
query91	170	101	101	101
query92	67	51	51	51
query93	6245	506	501	501
query94	1123	190	191	190
query95	410	323	320	320
query96	638	276	260	260
query97	3189	2960	2989	2960
query98	223	200	204	200
query99	1136	838	833	833
Total cold run time: 286639 ms
Total hot run time: 172862 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.95 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9870a88e0cc07a984b98ed5b3ae62fe05e5e7333, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.06
query4	1.67	0.11	0.10
query5	0.49	0.52	0.46
query6	1.13	0.72	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.56	0.49	0.49
query10	0.55	0.54	0.56
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.60	0.58	0.57
query14	0.77	0.78	0.78
query15	0.85	0.82	0.81
query16	0.36	0.35	0.37
query17	1.02	0.96	1.05
query18	0.22	0.25	0.24
query19	1.85	1.69	1.70
query20	0.01	0.01	0.01
query21	15.43	0.78	0.65
query22	4.02	6.52	2.27
query23	18.33	1.41	1.32
query24	2.13	0.23	0.24
query25	0.16	0.09	0.09
query26	0.27	0.17	0.18
query27	0.08	0.08	0.08
query28	13.20	1.01	1.00
query29	12.58	3.26	3.27
query30	0.25	0.06	0.06
query31	2.85	0.38	0.39
query32	3.28	0.48	0.47
query33	2.88	2.89	2.90
query34	17.10	4.45	4.51
query35	4.47	4.44	4.51
query36	0.65	0.46	0.48
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.14
query41	0.09	0.04	0.05
query42	0.06	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.24 s
Total hot run time: 30.95 s

@luwei16
Copy link
Contributor Author

luwei16 commented Jul 2, 2024

run p0

dataroaring
dataroaring approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 6969ad0 into apache:master Jul 2, 2024
25 of 29 checks passed
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Jul 8, 2024
@luwei16
[enhancement](compaction) optimizing memory usage for compaction (apa…
0354494 
…che#37099)
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@luwei16 @dataroaring
[enhancement](compaction) optimizing memory usage for compaction (#37099
f8bf9cc 
)
dataroaring pushed a commit that referenced this pull request Aug 4, 2024
@luwei16
[enhancement](compaction) optimizing memory usage for compaction (#37099
0603ec1 
) (#37486)
@dataroaring dataroaring mentioned this pull request Aug 4, 2024
Merged
@dataroaring dataroaring mentioned this pull request Aug 12, 2024
Merged
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring approved these changes

@gavinchou gavinchou gavinchou approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@luwei16 @doris-robot @gavinchou @dataroaring