Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache #38761

Merged
merged 2 commits into from
Aug 4, 2024
Merged

[Enhancement](merge-on-write) Support dynamic delete bitmap txn_delete_bitmap_cache #38761

merged 2 commits into from
Aug 4, 2024

Conversation

hust-hhb
Copy link
Contributor

@hust-hhb hust-hhb commented Aug 2, 2024

The default cache is set to 100MB, use memory limit to dynamic adjustmen

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@hust-hhb
Copy link
Contributor Author

hust-hhb commented Aug 2, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41272 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 562ba6da38f15f48731fc2065ff3de4fce78590d, data reload: false

------ Round 1 ----------------------------------
q1	17681	4019	4034	4019
q2	2014	198	194	194
q3	10454	1331	1352	1331
q4	10170	822	891	822
q5	7642	2948	2928	2928
q6	220	139	137	137
q7	1059	596	624	596
q8	9424	1885	1913	1885
q9	8512	6610	6554	6554
q10	8698	3812	3784	3784
q11	439	240	250	240
q12	401	228	231	228
q13	17771	2952	2976	2952
q14	274	238	247	238
q15	519	484	492	484
q16	533	387	383	383
q17	950	866	913	866
q18	7992	7305	7187	7187
q19	1392	1215	1205	1205
q20	551	325	337	325
q21	5252	4780	4636	4636
q22	344	278	279	278
Total cold run time: 112292 ms
Total hot run time: 41272 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4023	4010	3979	3979
q2	329	218	216	216
q3	2998	2980	3047	2980
q4	1981	1987	2002	1987
q5	5481	5562	5443	5443
q6	237	140	135	135
q7	2112	1795	1790	1790
q8	3286	3363	3299	3299
q9	8646	8585	8681	8585
q10	3938	3993	3974	3974
q11	537	457	453	453
q12	760	612	638	612
q13	16350	3153	3134	3134
q14	313	259	275	259
q15	543	489	478	478
q16	482	431	400	400
q17	1740	1727	1704	1704
q18	8123	7656	7778	7656
q19	1706	1685	1695	1685
q20	2073	1876	1832	1832
q21	5659	5349	5350	5349
q22	521	458	485	458
Total cold run time: 71838 ms
Total hot run time: 56408 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170247 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 562ba6da38f15f48731fc2065ff3de4fce78590d, data reload: false

query1	924	369	370	369
query2	6463	1739	1711	1711
query3	6675	209	218	209
query4	20022	17454	17357	17357
query5	3643	509	509	509
query6	276	170	166	166
query7	4592	303	300	300
query8	247	224	200	200
query9	8503	2370	2364	2364
query10	444	302	275	275
query11	10532	10124	9999	9999
query12	130	90	89	89
query13	1636	393	379	379
query14	9910	7395	6972	6972
query15	212	171	166	166
query16	6828	455	432	432
query17	945	586	575	575
query18	1937	297	329	297
query19	186	143	145	143
query20	94	88	86	86
query21	203	101	103	101
query22	4248	4102	4001	4001
query23	33759	33746	33490	33490
query24	10002	3091	3042	3042
query25	682	398	410	398
query26	1697	154	165	154
query27	2812	290	286	286
query28	7252	2014	2003	2003
query29	1287	440	449	440
query30	248	154	157	154
query31	958	752	775	752
query32	101	52	56	52
query33	693	328	338	328
query34	908	500	510	500
query35	865	756	765	756
query36	1036	888	924	888
query37	209	79	81	79
query38	2967	2827	2868	2827
query39	905	855	815	815
query40	265	112	114	112
query41	66	44	44	44
query42	123	106	105	105
query43	462	418	428	418
query44	1170	754	741	741
query45	208	175	173	173
query46	1085	812	818	812
query47	1789	1693	1729	1693
query48	368	295	293	293
query49	944	440	431	431
query50	897	450	445	445
query51	6848	6694	6693	6693
query52	113	88	94	88
query53	270	188	186	186
query54	601	456	460	456
query55	81	77	79	77
query56	275	250	261	250
query57	1144	1067	1064	1064
query58	295	279	300	279
query59	2605	2385	2359	2359
query60	304	305	274	274
query61	102	95	94	94
query62	856	663	664	663
query63	221	187	184	184
query64	5608	1910	1896	1896
query65	3164	3085	3111	3085
query66	1349	334	335	334
query67	15262	14892	15021	14892
query68	4359	576	586	576
query69	450	307	295	295
query70	1129	1103	1093	1093
query71	414	291	286	286
query72	7032	2698	2468	2468
query73	760	336	337	336
query74	5965	5682	5665	5665
query75	3393	2739	2762	2739
query76	2340	1236	1275	1236
query77	415	306	318	306
query78	9377	8959	8939	8939
query79	2665	535	537	535
query80	1148	544	518	518
query81	563	231	220	220
query82	1070	138	128	128
query83	235	190	171	171
query84	253	137	85	85
query85	1277	324	300	300
query86	454	302	332	302
query87	3212	3114	3125	3114
query88	3678	2504	2512	2504
query89	379	307	294	294
query90	1729	194	198	194
query91	128	101	103	101
query92	62	51	53	51
query93	2064	609	614	609
query94	845	294	308	294
query95	384	270	272	270
query96	615	292	283	283
query97	3233	3055	3136	3055
query98	227	253	197	197
query99	1633	1278	1319	1278
Total cold run time: 262994 ms
Total hot run time: 170247 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 562ba6da38f15f48731fc2065ff3de4fce78590d, data reload: false

query1	0.04	0.04	0.04
query2	0.07	0.03	0.03
query3	0.22	0.05	0.06
query4	1.68	0.07	0.07
query5	0.51	0.48	0.48
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.58	0.51	0.52
query10	0.57	0.56	0.56
query11	0.15	0.11	0.12
query12	0.15	0.13	0.13
query13	0.62	0.60	0.60
query14	0.77	0.79	0.81
query15	0.88	0.86	0.86
query16	0.36	0.36	0.37
query17	1.01	1.05	1.01
query18	0.22	0.20	0.20
query19	1.82	1.72	1.75
query20	0.02	0.01	0.01
query21	15.38	0.76	0.66
query22	4.28	6.90	1.90
query23	18.03	1.33	1.33
query24	2.27	0.22	0.22
query25	0.19	0.08	0.08
query26	0.31	0.21	0.21
query27	0.46	0.22	0.23
query28	13.17	1.02	0.96
query29	12.52	3.29	3.28
query30	0.25	0.06	0.06
query31	2.86	0.40	0.40
query32	3.26	0.48	0.49
query33	2.95	3.00	2.98
query34	15.46	4.26	4.23
query35	4.28	4.30	4.29
query36	0.67	0.48	0.48
query37	0.18	0.17	0.17
query38	0.17	0.16	0.15
query39	0.04	0.04	0.04
query40	0.16	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.97 s
Total hot run time: 30.71 s

zhannngchen
zhannngchen approved these changes Aug 2, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 2, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 2, 2024

PR approved by anyone and no changes requested.

@zhannngchen zhannngchen added p0_l and removed approved Indicates a PR has been approved by one committer. reviewed labels Aug 2, 2024
dataroaring
dataroaring approved these changes Aug 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 4, 2024

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 6619b64 into apache:master Aug 4, 2024
29 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Aug 4, 2024
@hust-hhb @dataroaring
[Enhancement](merge-on-write) Support dynamic delete bitmap txn_delet…
afe4ab7 
…e_bitmap_cache (#38761)

The default cache is set to 100MB, use memory limit to dynamic adjustmen
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged doing p0_l reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hust-hhb @doris-robot @zhannngchen @dataroaring