[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks #37318

luwei16 · 2024-07-04T15:01:10Z

Specify the specific stages of compaction tasks and accurately count concurrent tasks only while they are actively running

…action tasks

doris-robot · 2024-07-04T15:01:16Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions

clang-tidy made some suggestions

github-actions · 2024-07-04T15:08:18Z

be/test/olap/compaction_task_test.cpp

+// specific language governing permissions and limitations
+// under the License.
+
+#include <gmock/gmock-actions.h>


warning: 'gmock/gmock-actions.h' file not found [clang-diagnostic-error]

#include <gmock/gmock-actions.h> ^

gavinchou · 2024-07-04T15:16:52Z

be/src/olap/base_tablet.h

@@ -42,6 +43,8 @@ struct TabletWithVersion {
    int64_t version;
 };

+enum CompactionStage { NOT_SCHEDULED, PENDING, EXECUTING };


…paction-concurrent-num

luwei16 · 2024-07-05T02:34:43Z

run buildall

doris-robot · 2024-07-05T03:32:39Z

TPC-H: Total hot run time: 39638 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false

------ Round 1 ----------------------------------
q1	17606	4402	4276	4276
q2	2024	196	193	193
q3	10491	1184	1160	1160
q4	10203	794	772	772
q5	7480	2703	2593	2593
q6	222	135	133	133
q7	952	607	622	607
q8	9238	2062	2111	2062
q9	8841	6492	6492	6492
q10	8836	3632	3682	3632
q11	461	240	242	240
q12	435	233	224	224
q13	19070	2961	2975	2961
q14	265	243	235	235
q15	528	496	489	489
q16	502	375	373	373
q17	964	587	645	587
q18	7989	7477	7458	7458
q19	7112	1409	1342	1342
q20	660	326	324	324
q21	4892	3149	3225	3149
q22	381	344	336	336
Total cold run time: 119152 ms
Total hot run time: 39638 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4383	4215	4223	4215
q2	363	265	261	261
q3	2938	2926	2896	2896
q4	2057	1661	1754	1661
q5	5583	5500	5443	5443
q6	237	132	127	127
q7	2209	1904	1867	1867
q8	3297	3404	3438	3404
q9	8679	8721	8789	8721
q10	4101	3801	3690	3690
q11	584	516	499	499
q12	828	635	650	635
q13	15967	3130	3126	3126
q14	309	291	269	269
q15	541	474	475	474
q16	472	428	418	418
q17	1831	1516	1505	1505
q18	8009	8060	7980	7980
q19	1775	1481	1594	1481
q20	2166	1894	1865	1865
q21	5105	4981	4826	4826
q22	634	555	583	555
Total cold run time: 72068 ms
Total hot run time: 55918 ms

doris-robot · 2024-07-05T03:44:14Z

TPC-DS: Total hot run time: 174571 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false

query1	931	374	364	364
query2	6458	2540	2433	2433
query3	6641	203	211	203
query4	28630	17533	17040	17040
query5	3674	493	498	493
query6	260	165	183	165
query7	4590	300	301	300
query8	325	303	294	294
query9	8634	2521	2483	2483
query10	596	307	287	287
query11	12679	9997	10028	9997
query12	129	92	85	85
query13	1655	379	385	379
query14	10328	7813	7958	7813
query15	242	188	192	188
query16	7090	331	320	320
query17	1767	566	560	560
query18	1684	285	282	282
query19	201	161	155	155
query20	92	82	82	82
query21	214	129	128	128
query22	4199	4211	3978	3978
query23	34037	33461	33594	33461
query24	11143	2826	2848	2826
query25	644	421	390	390
query26	1128	173	166	166
query27	2421	323	344	323
query28	7277	2226	2205	2205
query29	907	662	644	644
query30	263	165	161	161
query31	958	772	747	747
query32	96	53	61	53
query33	766	326	310	310
query34	961	507	491	491
query35	746	647	641	641
query36	1131	996	999	996
query37	160	79	86	79
query38	2975	2823	2780	2780
query39	888	877	836	836
query40	219	138	127	127
query41	55	56	53	53
query42	111	99	105	99
query43	602	561	540	540
query44	1229	727	726	726
query45	197	162	165	162
query46	1069	748	749	748
query47	1878	1779	1801	1779
query48	367	401	296	296
query49	859	405	415	405
query50	754	387	383	383
query51	6879	6758	6798	6758
query52	100	98	95	95
query53	367	286	296	286
query54	921	454	450	450
query55	78	72	73	72
query56	283	265	268	265
query57	1120	1073	1037	1037
query58	252	235	255	235
query59	3591	3610	3340	3340
query60	304	277	273	273
query61	100	98	95	95
query62	627	454	439	439
query63	319	292	292	292
query64	8910	2219	1803	1803
query65	3165	3090	3145	3090
query66	765	341	332	332
query67	15749	15147	15020	15020
query68	8495	558	568	558
query69	743	452	342	342
query70	1248	1171	1104	1104
query71	535	287	270	270
query72	8851	6287	5539	5539
query73	1793	324	325	324
query74	6036	5483	5533	5483
query75	5190	2679	2662	2662
query76	4879	991	1009	991
query77	773	305	293	293
query78	9514	8970	8837	8837
query79	9232	521	524	521
query80	1190	473	489	473
query81	580	226	225	225
query82	739	109	110	109
query83	327	190	163	163
query84	273	91	85	85
query85	1422	299	298	298
query86	397	327	314	314
query87	3278	3233	3132	3132
query88	4086	2457	2489	2457
query89	542	384	382	382
query90	2051	191	192	191
query91	132	103	103	103
query92	65	48	49	48
query93	6547	530	518	518
query94	1284	213	210	210
query95	408	311	329	311
query96	615	273	269	269
query97	3200	2995	3061	2995
query98	219	196	196	196
query99	1182	844	830	830
Total cold run time: 304216 ms
Total hot run time: 174571 ms

doris-robot · 2024-07-05T03:49:48Z

ClickBench: Total hot run time: 31.13 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2749161ed51f74c10ab77dbb881dad3867408001, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.22	0.04	0.05
query4	1.67	0.07	0.07
query5	0.50	0.49	0.51
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.54	0.53	0.57
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.58
query14	0.77	0.78	0.78
query15	0.85	0.81	0.81
query16	0.36	0.37	0.35
query17	1.05	0.99	0.96
query18	0.25	0.22	0.27
query19	1.94	1.77	1.82
query20	0.01	0.01	0.02
query21	15.44	0.75	0.64
query22	4.16	7.22	2.38
query23	18.26	1.42	1.31
query24	2.12	0.24	0.22
query25	0.15	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.28	1.02	0.99
query29	12.61	3.31	3.32
query30	0.25	0.06	0.06
query31	2.86	0.40	0.40
query32	3.25	0.47	0.48
query33	2.91	2.89	2.92
query34	17.08	4.31	4.32
query35	4.39	4.44	4.39
query36	0.66	0.48	0.48
query37	0.19	0.16	0.15
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.14	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 31.13 s

dataroaring

LGTM

github-actions · 2024-07-05T14:26:14Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-07-05T14:26:15Z

PR approved by anyone and no changes requested.

…action tasks (apache#37318) Specify the specific stages of compaction tasks and accurately count concurrent tasks only while they are actively running

fix compilation error introduced by #37165 and #37318

…action tasks (#37318) (#37496)

…action tasks (#37318) Specify the specific stages of compaction tasks and accurately count concurrent tasks only while they are actively running

fix compilation error introduced by #37165 and #37318

[fix](compaction) fixing the inaccurate statistics of concurrent comp…

92cb73a

…action tasks

github-actions bot reviewed Jul 4, 2024

View reviewed changes

gavinchou reviewed Jul 4, 2024

View reviewed changes

luwei16 added 2 commits July 5, 2024 10:10

Merge remote-tracking branch 'doris-origin/master' into luwei/fix-com…

533fd99

…paction-concurrent-num

fix

2749161

dataroaring added the dev/2.1.x label Jul 5, 2024

dataroaring approved these changes Jul 5, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024

github-actions bot added the reviewed label Jul 5, 2024

gavinchou approved these changes Jul 5, 2024

View reviewed changes

gavinchou merged commit 5641c77 into apache:master Jul 5, 2024
25 of 29 checks passed

gavinchou mentioned this pull request Jul 9, 2024

[fix](Compile) Use correct sync point header file #37548

Merged

hello-stephen pushed a commit that referenced this pull request Jul 9, 2024

[fix](Compile) Use correct sync point header file (#37548)

69372ad

fix compilation error introduced by #37165 and #37318

dataroaring pushed a commit that referenced this pull request Jul 10, 2024

[fix](compaction) fixing the inaccurate statistics of concurrent comp…

9f4e734

…action tasks (#37318) (#37496)

dataroaring mentioned this pull request Jul 10, 2024

[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks (#37318) #37496

Merged

dataroaring added dev/2.1.5-merged and removed dev/2.1.x labels Jul 10, 2024

dataroaring pushed a commit that referenced this pull request Jul 17, 2024

[fix](Compile) Use correct sync point header file (#37548)

89b6e27

fix compilation error introduced by #37165 and #37318

dataroaring added the dev/3.0.1-merged label Aug 5, 2024

gavinchou mentioned this pull request Aug 19, 2024

Release Note 3.0.1 #39570

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks #37318

[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks #37318

luwei16 commented Jul 4, 2024 •

edited

Loading

doris-robot commented Jul 4, 2024

github-actions bot left a comment

github-actions bot Jul 4, 2024

gavinchou Jul 4, 2024

luwei16 commented Jul 5, 2024

doris-robot commented Jul 5, 2024

doris-robot commented Jul 5, 2024

doris-robot commented Jul 5, 2024

dataroaring left a comment

github-actions bot commented Jul 5, 2024

github-actions bot commented Jul 5, 2024

[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks #37318

[fix](compaction) fixing the inaccurate statistics of concurrent compaction tasks #37318

Conversation

luwei16 commented Jul 4, 2024 • edited Loading

doris-robot commented Jul 4, 2024

github-actions bot left a comment

Choose a reason for hiding this comment

github-actions bot Jul 4, 2024

Choose a reason for hiding this comment

gavinchou Jul 4, 2024

Choose a reason for hiding this comment

luwei16 commented Jul 5, 2024

doris-robot commented Jul 5, 2024

doris-robot commented Jul 5, 2024

doris-robot commented Jul 5, 2024

dataroaring left a comment

Choose a reason for hiding this comment

github-actions bot commented Jul 5, 2024

github-actions bot commented Jul 5, 2024

luwei16 commented Jul 4, 2024 •

edited

Loading