Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](meta) put normal partitions into recyclebin when replacing partitions #36361

Merged
merged 11 commits into from
Jul 8, 2024
Merged

[opt](meta) put normal partitions into recyclebin when replacing partitions #36361

merged 11 commits into from
Jul 8, 2024

Conversation

liutang123
Copy link
Contributor

@liutang123 liutang123 commented Jun 16, 2024 •
edited
Loading

Proposed changes

Issue Number: close #36714

Move the old partitions to recycle bin when replacing partition.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@dataroaring
Copy link
Contributor

Please handle truncate too, and add regression test.

@dataroaring dataroaring changed the title [opt](meta) Recycle partition when replace partition [opt](meta) put old partitions into recyclebin when replace partition Jun 17, 2024
@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39566 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 862df4fa7f2afbf07ff971cbc3bea728c665b117, data reload: false

------ Round 1 ----------------------------------
q1	17616	4293	4271	4271
q2	2034	194	200	194
q3	10455	1144	1154	1144
q4	10199	778	795	778
q5	7456	2614	2615	2614
q6	219	137	136	136
q7	970	607	597	597
q8	9233	2044	2049	2044
q9	9022	6447	6448	6447
q10	8943	3750	3701	3701
q11	467	240	248	240
q12	475	243	229	229
q13	17779	2962	2965	2962
q14	275	226	233	226
q15	498	485	470	470
q16	523	379	383	379
q17	947	665	662	662
q18	7959	7417	7284	7284
q19	6507	1469	1427	1427
q20	664	321	324	321
q21	4965	3292	3100	3100
q22	390	345	340	340
Total cold run time: 117596 ms
Total hot run time: 39566 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4432	4213	4362	4213
q2	363	274	285	274
q3	2955	2869	2945	2869
q4	1951	1731	1689	1689
q5	5639	5516	5486	5486
q6	213	131	133	131
q7	2227	1937	1836	1836
q8	3257	3402	3419	3402
q9	8721	8653	8807	8653
q10	4065	3898	3739	3739
q11	574	509	511	509
q12	813	632	629	629
q13	15918	3206	3150	3150
q14	311	281	273	273
q15	532	484	493	484
q16	499	438	423	423
q17	1799	1496	1498	1496
q18	7986	7947	7757	7757
q19	1782	1527	1597	1527
q20	3026	1924	1895	1895
q21	9225	4970	4879	4879
q22	638	545	567	545
Total cold run time: 76926 ms
Total hot run time: 55859 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173985 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 862df4fa7f2afbf07ff971cbc3bea728c665b117, data reload: false

query1	925	380	373	373
query2	6461	2441	2352	2352
query3	6630	202	204	202
query4	19191	17354	17233	17233
query5	3581	496	509	496
query6	240	161	158	158
query7	4592	304	293	293
query8	318	298	281	281
query9	8539	2383	2380	2380
query10	565	299	283	283
query11	10585	10008	10106	10008
query12	123	92	87	87
query13	1647	377	356	356
query14	9566	7091	7844	7091
query15	255	185	189	185
query16	7753	271	260	260
query17	1705	539	523	523
query18	1950	281	276	276
query19	208	156	169	156
query20	99	92	89	89
query21	214	129	129	129
query22	4260	4040	4039	4039
query23	33895	33792	33839	33792
query24	11054	2890	3000	2890
query25	620	393	378	378
query26	1251	165	162	162
query27	2506	325	336	325
query28	7186	2099	2104	2099
query29	887	624	658	624
query30	246	160	159	159
query31	963	750	762	750
query32	95	54	59	54
query33	786	331	278	278
query34	1055	501	507	501
query35	746	619	641	619
query36	1124	943	958	943
query37	152	71	71	71
query38	2943	2841	2805	2805
query39	926	857	846	846
query40	207	134	133	133
query41	63	54	53	53
query42	139	104	108	104
query43	608	558	565	558
query44	1311	730	740	730
query45	193	161	164	161
query46	1067	710	737	710
query47	1886	1764	1740	1740
query48	380	299	289	289
query49	857	401	420	401
query50	779	405	389	389
query51	6662	6709	6659	6659
query52	110	92	95	92
query53	360	300	297	297
query54	923	445	443	443
query55	76	76	73	73
query56	287	261	263	261
query57	1140	1028	1030	1028
query58	243	257	249	249
query59	3402	3262	3282	3262
query60	299	270	282	270
query61	96	86	87	86
query62	593	456	473	456
query63	318	294	298	294
query64	8829	2300	1829	1829
query65	3190	3079	3122	3079
query66	735	326	324	324
query67	15274	14891	14932	14891
query68	4502	543	555	543
query69	550	416	378	378
query70	1189	1101	1172	1101
query71	426	273	278	273
query72	7049	5527	5262	5262
query73	770	327	328	327
query74	5902	5518	5494	5494
query75	3399	2672	2719	2672
query76	2397	929	965	929
query77	477	305	299	299
query78	10244	9997	9672	9672
query79	2272	524	509	509
query80	2246	480	464	464
query81	605	218	223	218
query82	1067	109	102	102
query83	317	176	169	169
query84	263	89	84	84
query85	1257	286	271	271
query86	464	324	332	324
query87	3274	3141	3063	3063
query88	3025	2350	2341	2341
query89	483	412	392	392
query90	1692	197	190	190
query91	129	98	104	98
query92	60	51	49	49
query93	1972	510	514	510
query94	1133	190	185	185
query95	400	327	320	320
query96	595	263	269	263
query97	3247	3093	3062	3062
query98	227	205	197	197
query99	1383	817	841	817
Total cold run time: 268885 ms
Total hot run time: 173985 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 862df4fa7f2afbf07ff971cbc3bea728c665b117, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.05
query3	0.22	0.05	0.04
query4	1.69	0.07	0.07
query5	0.50	0.49	0.50
query6	1.13	0.72	0.72
query7	0.02	0.01	0.02
query8	0.04	0.04	0.04
query9	0.54	0.49	0.50
query10	0.54	0.56	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.60	0.60	0.61
query14	0.77	0.77	0.78
query15	0.82	0.81	0.81
query16	0.36	0.36	0.35
query17	0.97	1.01	1.00
query18	0.21	0.27	0.24
query19	1.76	1.69	1.71
query20	0.01	0.01	0.02
query21	15.52	0.66	0.65
query22	3.98	7.88	1.76
query23	18.33	1.36	1.24
query24	2.15	0.21	0.21
query25	0.16	0.07	0.07
query26	0.27	0.17	0.18
query27	0.08	0.09	0.07
query28	13.27	1.02	1.01
query29	12.66	3.29	3.24
query30	0.27	0.06	0.06
query31	2.87	0.40	0.39
query32	3.27	0.46	0.47
query33	2.89	2.88	2.84
query34	17.04	4.46	4.49
query35	4.49	4.51	4.50
query36	0.64	0.49	0.46
query37	0.18	0.16	0.17
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.15	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.22 s
Total hot run time: 30.35 s

@dataroaring dataroaring changed the title [opt](meta) put old partitions into recyclebin when replace partition [opt](meta) put old partitions into recyclebin when replacing partitions Jun 17, 2024
@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from e3add86 to 43ad8ce Compare June 17, 2024 18:21
@liutang123
Copy link
Contributor Author

run buildall

@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from 43ad8ce to 13bb106 Compare June 19, 2024 15:56
@liutang123
Copy link
Contributor Author

run buildall

@liutang123 liutang123 changed the title [opt](meta) put old partitions into recyclebin when replacing partitions [opt](meta) put normal partitions into recyclebin when replacing partitions Jun 19, 2024
@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from 13bb106 to 43ab7fd Compare June 19, 2024 16:17
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39727 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 43ab7fdffe24d45d5a7f90bc1a38a0047a0cf9e5, data reload: false

------ Round 1 ----------------------------------
q1	4986	4738	4317	4317
q2	1623	204	206	204
q3	9101	1091	1096	1091
q4	10554	877	743	743
q5	7463	2693	2609	2609
q6	221	137	137	137
q7	964	628	613	613
q8	9302	2077	2049	2049
q9	9631	6488	6465	6465
q10	9009	3739	3732	3732
q11	458	246	253	246
q12	431	235	239	235
q13	19012	2950	3023	2950
q14	273	214	223	214
q15	519	464	478	464
q16	495	387	375	375
q17	991	671	672	671
q18	8208	7437	7295	7295
q19	6002	1399	1528	1399
q20	659	312	327	312
q21	4945	3260	3801	3260
q22	398	346	351	346
Total cold run time: 105245 ms
Total hot run time: 39727 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4375	4262	4210	4210
q2	369	259	265	259
q3	2988	2731	2738	2731
q4	1885	1601	1680	1601
q5	5255	5298	5278	5278
q6	222	125	125	125
q7	2132	1746	1793	1746
q8	3208	3331	3312	3312
q9	8361	8328	8347	8328
q10	3879	3704	3704	3704
q11	582	477	469	469
q12	766	576	631	576
q13	17327	2987	3024	2987
q14	302	254	267	254
q15	509	475	471	471
q16	463	409	413	409
q17	1767	1486	1494	1486
q18	7709	7581	7460	7460
q19	1735	1671	1584	1584
q20	2000	1802	1772	1772
q21	4786	4787	4667	4667
q22	604	536	532	532
Total cold run time: 71224 ms
Total hot run time: 53961 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171893 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 43ab7fdffe24d45d5a7f90bc1a38a0047a0cf9e5, data reload: false

query1	929	391	381	381
query2	6453	2327	2409	2327
query3	6649	211	210	210
query4	19973	17364	17277	17277
query5	4138	487	464	464
query6	249	161	153	153
query7	4590	309	298	298
query8	345	307	284	284
query9	8504	2404	2394	2394
query10	614	296	302	296
query11	10534	10087	10039	10039
query12	140	96	86	86
query13	1645	369	379	369
query14	10225	6316	7495	6316
query15	230	197	187	187
query16	7835	261	263	261
query17	1874	553	518	518
query18	1962	269	273	269
query19	204	157	151	151
query20	89	92	82	82
query21	215	134	128	128
query22	4418	4194	4051	4051
query23	33708	33001	33124	33001
query24	11980	2860	2801	2801
query25	666	351	363	351
query26	1821	152	147	147
query27	3070	323	327	323
query28	7221	2056	2044	2044
query29	1145	635	621	621
query30	277	154	150	150
query31	948	725	751	725
query32	100	54	54	54
query33	763	289	296	289
query34	956	467	464	464
query35	747	630	610	610
query36	1081	963	930	930
query37	197	72	68	68
query38	2877	2749	2753	2749
query39	854	796	795	795
query40	285	129	128	128
query41	58	53	55	53
query42	119	97	101	97
query43	595	546	562	546
query44	1197	724	743	724
query45	200	165	160	160
query46	1093	732	699	699
query47	1857	1773	1799	1773
query48	374	302	291	291
query49	1201	405	443	405
query50	775	394	389	389
query51	6842	6734	6618	6618
query52	114	86	93	86
query53	357	291	287	287
query54	904	434	438	434
query55	75	73	71	71
query56	293	256	266	256
query57	1135	1035	1080	1035
query58	253	250	253	250
query59	3382	3150	3155	3150
query60	302	295	264	264
query61	92	89	93	89
query62	672	448	445	445
query63	320	297	295	295
query64	9925	2245	1745	1745
query65	3174	3248	3122	3122
query66	1358	354	341	341
query67	15376	14847	14850	14847
query68	4581	542	535	535
query69	604	469	353	353
query70	1171	1158	1135	1135
query71	443	265	271	265
query72	7717	5497	5320	5320
query73	748	327	319	319
query74	5926	5574	5552	5552
query75	3643	2669	2688	2669
query76	2981	973	881	881
query77	648	301	294	294
query78	10291	9968	9683	9683
query79	2441	526	519	519
query80	1092	462	454	454
query81	563	226	228	226
query82	1437	104	103	103
query83	267	169	169	169
query84	236	86	85	85
query85	1588	286	333	286
query86	483	324	337	324
query87	3234	3143	3090	3090
query88	3826	2343	2343	2343
query89	485	378	381	378
query90	1805	191	194	191
query91	129	99	99	99
query92	65	48	50	48
query93	2392	518	496	496
query94	1252	186	183	183
query95	396	305	316	305
query96	600	267	266	266
query97	3240	3030	3070	3030
query98	208	198	203	198
query99	1201	857	831	831
Total cold run time: 278247 ms
Total hot run time: 171893 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 43ab7fdffe24d45d5a7f90bc1a38a0047a0cf9e5, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.04	0.06
query4	1.67	0.07	0.08
query5	0.51	0.53	0.50
query6	1.15	0.74	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.54	0.50	0.50
query10	0.55	0.55	0.53
query11	0.16	0.12	0.11
query12	0.14	0.12	0.13
query13	0.59	0.59	0.60
query14	0.79	0.79	0.78
query15	0.83	0.82	0.84
query16	0.37	0.37	0.37
query17	1.01	1.05	1.06
query18	0.22	0.23	0.25
query19	1.78	1.81	1.68
query20	0.02	0.01	0.01
query21	15.41	0.66	0.63
query22	4.11	7.15	2.22
query23	18.32	1.42	1.24
query24	2.19	0.23	0.22
query25	0.16	0.09	0.07
query26	0.26	0.18	0.17
query27	0.09	0.07	0.08
query28	13.24	1.02	1.01
query29	12.64	3.31	3.31
query30	0.26	0.06	0.06
query31	2.88	0.40	0.40
query32	3.28	0.48	0.48
query33	2.90	2.88	2.92
query34	17.18	4.36	4.42
query35	4.45	4.42	4.50
query36	0.65	0.48	0.46
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.04	0.04
query40	0.16	0.14	0.15
query41	0.10	0.05	0.04
query42	0.06	0.06	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.5 s
Total hot run time: 30.83 s

@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from 253f055 to 4f6c77a Compare June 20, 2024 01:08
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40320 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4f6c77a93c6eda798d23971d98e84df3c404e384, data reload: false

------ Round 1 ----------------------------------
q1	18658	4326	4293	4293
q2	2031	195	197	195
q3	10441	1099	1121	1099
q4	10496	857	869	857
q5	8487	2683	2684	2683
q6	220	134	134	134
q7	958	616	601	601
q8	9216	2059	2088	2059
q9	9031	6579	6614	6579
q10	9148	3829	3935	3829
q11	456	251	250	250
q12	443	251	240	240
q13	18935	3052	3058	3052
q14	270	216	239	216
q15	525	489	489	489
q16	689	410	388	388
q17	1371	696	747	696
q18	8128	7425	7400	7400
q19	5149	1451	1405	1405
q20	646	326	330	326
q21	4915	3192	3886	3192
q22	408	337	337	337
Total cold run time: 120621 ms
Total hot run time: 40320 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4278	4242	4242
q2	420	278	273	273
q3	3145	2871	2920	2871
q4	1995	1762	1740	1740
q5	5515	5475	5515	5475
q6	222	132	133	132
q7	2203	1832	1825	1825
q8	3307	3397	3389	3389
q9	9139	9354	9095	9095
q10	4002	3772	3808	3772
q11	607	487	497	487
q12	808	647	631	631
q13	16302	3192	3202	3192
q14	302	280	259	259
q15	538	479	472	472
q16	487	450	427	427
q17	1808	1521	1493	1493
q18	8200	7834	7903	7834
q19	1788	1572	1494	1494
q20	2580	1892	1838	1838
q21	5214	4669	4845	4669
q22	631	587	535	535
Total cold run time: 73543 ms
Total hot run time: 56145 ms

@liutang123
Copy link
Contributor Author

run buildall

zddr
zddr approved these changes Jun 20, 2024
View reviewed changes
Copy link
Contributor

@zddr zddr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

dataroaring
dataroaring reviewed Jun 21, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/persist/ReplacePartitionOperationLog.java Outdated
@@ -53,11 +54,13 @@ public class ReplacePartitionOperationLog implements Writable {
private long version = 0L;
@SerializedName(value = "versionTime")
private long versionTime = 0L;
@SerializedName(value = "force")
private boolean force = FeConstants.DEFAULT_DROP_NORMAL_PARTITION_WHEN_REPLACE_PARTITION;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

default to false.

@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch 2 times, most recently from 00ae8f2 to bc0309d Compare June 23, 2024 11:25
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40035 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a73ffccfd7f218ce793e2d421514c08f84923707, data reload: false

------ Round 1 ----------------------------------
q1	17858	4477	4427	4427
q2	2591	198	200	198
q3	11793	1105	1148	1105
q4	10593	814	802	802
q5	7532	2715	2763	2715
q6	238	140	142	140
q7	982	640	629	629
q8	9527	2080	2039	2039
q9	8852	6430	6487	6430
q10	8997	3712	3708	3708
q11	451	242	230	230
q12	439	240	221	221
q13	17779	2980	2970	2970
q14	275	230	220	220
q15	514	481	475	475
q16	532	380	377	377
q17	974	752	647	647
q18	8109	7401	7353	7353
q19	7126	1438	1460	1438
q20	694	314	337	314
q21	4985	3269	3875	3269
q22	387	330	328	328
Total cold run time: 121228 ms
Total hot run time: 40035 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4390	4208	4218	4208
q2	379	269	263	263
q3	2990	2689	2722	2689
q4	1886	1626	1619	1619
q5	5253	5282	5274	5274
q6	213	125	131	125
q7	2160	1760	1771	1760
q8	3183	3303	3312	3303
q9	8350	8331	8356	8331
q10	3886	3652	3612	3612
q11	588	489	477	477
q12	766	585	611	585
q13	17346	2970	2973	2970
q14	279	264	250	250
q15	503	470	476	470
q16	479	426	412	412
q17	1757	1468	1475	1468
q18	7602	7575	7336	7336
q19	1685	1526	1588	1526
q20	1983	1767	1749	1749
q21	4881	4745	4633	4633
q22	587	504	534	504
Total cold run time: 71146 ms
Total hot run time: 53564 ms

@doris-robot
Copy link

TPC-H: Total hot run time: 39998 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46c7c619ad14113c905037e1dc99b9e2e0b1911b, data reload: false

------ Round 1 ----------------------------------
q1	18629	5299	4427	4427
q2	2616	192	193	192
q3	11535	1199	1077	1077
q4	10617	804	835	804
q5	8835	2717	2593	2593
q6	225	137	140	137
q7	952	629	590	590
q8	9225	2085	2056	2056
q9	8869	6457	6452	6452
q10	8986	3689	3742	3689
q11	455	229	227	227
q12	435	240	237	237
q13	17792	2983	2981	2981
q14	261	221	211	211
q15	528	463	486	463
q16	501	384	369	369
q17	956	742	791	742
q18	8078	7380	7365	7365
q19	7764	1559	1483	1483
q20	651	315	338	315
q21	4821	3258	3904	3258
q22	384	330	334	330
Total cold run time: 123115 ms
Total hot run time: 39998 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4222	4332	4222
q2	369	268	255	255
q3	2986	2749	2702	2702
q4	1857	1613	1588	1588
q5	5224	5281	5262	5262
q6	222	132	132	132
q7	2133	1698	1734	1698
q8	3185	3352	3298	3298
q9	8321	8305	8284	8284
q10	3880	3677	3608	3608
q11	564	480	482	480
q12	798	622	636	622
q13	17483	2970	2992	2970
q14	289	257	256	256
q15	515	488	481	481
q16	478	412	416	412
q17	1753	1454	1446	1446
q18	7626	7417	7363	7363
q19	1720	1602	1658	1602
q20	1995	1781	1757	1757
q21	4766	4627	4764	4627
q22	611	561	538	538
Total cold run time: 71211 ms
Total hot run time: 53603 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172633 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46c7c619ad14113c905037e1dc99b9e2e0b1911b, data reload: false

query1	925	396	368	368
query2	6470	2439	2391	2391
query3	6653	213	215	213
query4	18794	17623	17217	17217
query5	4245	483	493	483
query6	265	177	170	170
query7	4587	296	293	293
query8	321	291	306	291
query9	8480	2358	2331	2331
query10	613	300	288	288
query11	10499	10092	9937	9937
query12	129	85	87	85
query13	1656	363	386	363
query14	10147	7229	7483	7229
query15	235	189	184	184
query16	7956	262	283	262
query17	1877	539	534	534
query18	1991	274	272	272
query19	198	147	144	144
query20	91	80	81	80
query21	222	143	127	127
query22	4293	4103	4053	4053
query23	33804	33070	33135	33070
query24	12040	2840	2880	2840
query25	678	359	358	358
query26	1742	153	148	148
query27	3030	306	312	306
query28	7574	2016	2002	2002
query29	1031	592	598	592
query30	295	166	157	157
query31	957	750	745	745
query32	94	53	52	52
query33	768	278	281	278
query34	984	461	464	461
query35	753	631	641	631
query36	1046	938	932	932
query37	200	75	74	74
query38	2895	2732	2739	2732
query39	858	814	806	806
query40	271	120	121	120
query41	56	55	50	50
query42	117	96	107	96
query43	585	575	569	569
query44	1203	708	716	708
query45	188	161	158	158
query46	1093	712	701	701
query47	1823	1740	1764	1740
query48	362	295	292	292
query49	1204	411	415	411
query50	786	385	394	385
query51	6909	6807	6683	6683
query52	104	93	90	90
query53	356	297	300	297
query54	955	430	433	430
query55	76	72	73	72
query56	285	258	266	258
query57	1124	1006	1021	1006
query58	270	234	241	234
query59	3445	3350	3087	3087
query60	303	284	274	274
query61	93	89	102	89
query62	646	433	443	433
query63	315	293	292	292
query64	9858	2220	1791	1791
query65	3192	3097	3094	3094
query66	1410	336	333	333
query67	15455	14750	14897	14750
query68	5799	546	527	527
query69	590	412	335	335
query70	1217	1158	1134	1134
query71	445	274	279	274
query72	7476	5934	5333	5333
query73	780	323	323	323
query74	5890	5532	5610	5532
query75	3871	2668	2674	2668
query76	3673	875	854	854
query77	662	298	308	298
query78	10313	9819	9874	9819
query79	2233	515	508	508
query80	1768	486	515	486
query81	538	221	220	220
query82	798	109	104	104
query83	263	179	166	166
query84	263	91	83	83
query85	1430	275	276	275
query86	472	328	333	328
query87	3225	3092	3109	3092
query88	3885	2357	2390	2357
query89	468	385	381	381
query90	1819	189	189	189
query91	129	102	101	101
query92	60	47	48	47
query93	2457	491	489	489
query94	1167	194	198	194
query95	402	317	311	311
query96	598	267	263	263
query97	3170	2990	2990	2990
query98	224	200	186	186
query99	1139	824	841	824
Total cold run time: 279104 ms
Total hot run time: 172633 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.03 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46c7c619ad14113c905037e1dc99b9e2e0b1911b, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.03
query3	0.22	0.05	0.05
query4	1.68	0.06	0.07
query5	0.50	0.50	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.52	0.54	0.53
query11	0.15	0.12	0.11
query12	0.15	0.12	0.11
query13	0.59	0.59	0.58
query14	0.76	0.77	0.79
query15	0.85	0.82	0.81
query16	0.36	0.37	0.38
query17	0.96	1.02	0.94
query18	0.25	0.23	0.26
query19	1.78	1.71	1.67
query20	0.01	0.01	0.01
query21	15.45	0.78	0.66
query22	4.50	6.24	2.50
query23	18.22	1.44	1.32
query24	2.09	0.24	0.22
query25	0.14	0.09	0.09
query26	0.28	0.19	0.19
query27	0.08	0.07	0.07
query28	13.21	1.02	1.01
query29	12.59	3.27	3.27
query30	0.25	0.06	0.05
query31	2.86	0.40	0.37
query32	3.29	0.49	0.48
query33	2.90	2.91	2.85
query34	17.15	4.41	4.44
query35	4.51	4.45	4.45
query36	0.65	0.47	0.46
query37	0.18	0.16	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.16	0.15	0.14
query41	0.09	0.05	0.04
query42	0.05	0.04	0.04
query43	0.05	0.03	0.04
Total cold run time: 109.57 s
Total hot run time: 31.03 s

@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from 46c7c61 to b0526af Compare July 2, 2024 13:19
@liutang123
Copy link
Contributor Author

run p0

@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39903 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b0526af681aa8b09342d81faa8e0d8f7e28e50ad, data reload: false

------ Round 1 ----------------------------------
q1	17617	4469	4310	4310
q2	2025	192	181	181
q3	10513	1259	1180	1180
q4	10212	769	828	769
q5	7512	2691	2659	2659
q6	228	138	140	138
q7	962	609	612	609
q8	9243	2111	2090	2090
q9	8987	6511	6470	6470
q10	8973	3986	3719	3719
q11	468	233	240	233
q12	465	235	234	234
q13	17776	2973	2983	2973
q14	269	216	221	216
q15	518	484	477	477
q16	521	392	377	377
q17	970	707	748	707
q18	8244	7391	7474	7391
q19	6792	1494	1410	1410
q20	670	306	322	306
q21	4968	3114	4021	3114
q22	388	340	348	340
Total cold run time: 118321 ms
Total hot run time: 39903 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4478	4223	4298	4223
q2	370	265	263	263
q3	3022	2893	2943	2893
q4	1963	1732	1789	1732
q5	5598	5489	5411	5411
q6	222	130	139	130
q7	2259	1858	1851	1851
q8	3274	3447	3459	3447
q9	8763	8824	8787	8787
q10	4150	3647	3794	3647
q11	606	501	491	491
q12	842	666	629	629
q13	17066	3172	3204	3172
q14	307	276	293	276
q15	541	494	486	486
q16	509	432	445	432
q17	1836	1523	1469	1469
q18	8320	7946	7672	7672
q19	1771	1541	1655	1541
q20	2212	1875	1839	1839
q21	5097	4782	4746	4746
q22	600	556	570	556
Total cold run time: 73806 ms
Total hot run time: 55693 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170504 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b0526af681aa8b09342d81faa8e0d8f7e28e50ad, data reload: false

query1	910	369	370	369
query2	6439	2418	2265	2265
query3	6637	203	214	203
query4	19908	17657	17333	17333
query5	3647	496	492	492
query6	262	173	167	167
query7	4598	310	298	298
query8	309	287	297	287
query9	8531	2372	2363	2363
query10	562	297	290	290
query11	10725	10137	10182	10137
query12	124	89	82	82
query13	1634	371	373	371
query14	10163	6537	7562	6537
query15	237	185	182	182
query16	7917	271	260	260
query17	1909	539	516	516
query18	2024	270	261	261
query19	199	148	151	148
query20	89	81	88	81
query21	204	131	123	123
query22	4539	4415	4186	4186
query23	33968	33814	33488	33488
query24	11016	2906	2844	2844
query25	604	378	372	372
query26	1018	157	154	154
query27	2385	331	328	328
query28	6582	2151	2129	2129
query29	867	656	632	632
query30	240	156	154	154
query31	1013	758	750	750
query32	96	55	56	55
query33	778	308	298	298
query34	950	483	499	483
query35	746	631	642	631
query36	1182	989	959	959
query37	138	83	82	82
query38	2926	2838	2798	2798
query39	900	856	797	797
query40	206	127	127	127
query41	56	51	55	51
query42	119	101	100	100
query43	592	524	535	524
query44	1210	745	738	738
query45	226	172	168	168
query46	1078	726	733	726
query47	1848	1824	1810	1810
query48	367	313	309	309
query49	851	419	415	415
query50	774	385	396	385
query51	6896	6773	6788	6773
query52	106	90	91	90
query53	354	285	292	285
query54	912	448	448	448
query55	79	73	72	72
query56	289	277	272	272
query57	1119	1070	1050	1050
query58	248	242	253	242
query59	3286	3132	3213	3132
query60	312	278	277	277
query61	91	103	93	93
query62	587	438	452	438
query63	316	295	296	295
query64	8714	2261	1710	1710
query65	3184	3089	3058	3058
query66	745	318	323	318
query67	15633	14843	15161	14843
query68	5617	547	560	547
query69	646	455	327	327
query70	1196	1086	1157	1086
query71	472	285	281	281
query72	7020	2781	2582	2582
query73	768	326	329	326
query74	5950	5457	5452	5452
query75	3898	2682	2652	2652
query76	3636	909	879	879
query77	613	304	364	304
query78	10390	9774	9813	9774
query79	2374	520	518	518
query80	1154	464	472	464
query81	566	219	220	219
query82	1096	106	114	106
query83	198	164	171	164
query84	272	86	91	86
query85	1287	275	273	273
query86	463	323	290	290
query87	3292	3076	3042	3042
query88	3788	2469	2447	2447
query89	456	380	383	380
query90	1791	187	185	185
query91	127	101	100	100
query92	63	47	48	47
query93	2258	508	514	508
query94	1042	191	190	190
query95	406	314	325	314
query96	595	274	264	264
query97	3189	3007	3048	3007
query98	218	197	195	195
query99	1164	857	841	841
Total cold run time: 273179 ms
Total hot run time: 170504 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b0526af681aa8b09342d81faa8e0d8f7e28e50ad, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.09	0.09
query5	0.50	0.47	0.48
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.55	0.47	0.48
query10	0.54	0.53	0.55
query11	0.15	0.12	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.57
query14	0.76	0.78	0.78
query15	0.84	0.80	0.81
query16	0.36	0.38	0.35
query17	0.99	0.97	1.02
query18	0.24	0.25	0.25
query19	1.81	1.68	1.65
query20	0.01	0.01	0.01
query21	15.42	0.82	0.66
query22	4.57	7.37	1.70
query23	18.22	1.33	1.26
query24	2.12	0.24	0.21
query25	0.15	0.09	0.09
query26	0.26	0.18	0.18
query27	0.08	0.08	0.08
query28	13.25	1.03	0.99
query29	12.69	3.27	3.21
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.27	0.47	0.48
query33	2.90	2.91	2.89
query34	17.10	4.43	4.41
query35	4.42	4.46	4.50
query36	0.65	0.46	0.47
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.17	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.69 s
Total hot run time: 30.18 s

@liutang123 liutang123 force-pushed the opt-recycle-p-replace branch from b0526af to 35210d8 Compare July 3, 2024 01:16
@liutang123
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39766 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 35210d8492e7e5e528c8d4346dde19d270998514, data reload: false

------ Round 1 ----------------------------------
q1	17633	4270	4262	4262
q2	2006	192	183	183
q3	10477	1222	1117	1117
q4	10184	806	945	806
q5	7537	2655	2616	2616
q6	219	133	133	133
q7	941	600	614	600
q8	9224	2066	2051	2051
q9	8707	6481	6383	6383
q10	8993	3673	3728	3673
q11	479	233	234	233
q12	413	234	236	234
q13	17773	3027	2978	2978
q14	257	230	217	217
q15	530	489	473	473
q16	509	388	377	377
q17	950	596	722	596
q18	8050	7414	7416	7414
q19	6899	1533	1497	1497
q20	656	325	320	320
q21	4825	3824	3261	3261
q22	404	342	346	342
Total cold run time: 117666 ms
Total hot run time: 39766 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4508	4223	4768	4223
q2	372	265	259	259
q3	3006	2929	2948	2929
q4	2018	1687	1738	1687
q5	5562	5458	5431	5431
q6	221	136	140	136
q7	2220	1902	1805	1805
q8	3240	3434	3422	3422
q9	8713	8794	8860	8794
q10	4160	3900	3838	3838
q11	613	508	517	508
q12	813	650	661	650
q13	17113	3218	3259	3218
q14	312	302	274	274
q15	542	499	490	490
q16	510	454	425	425
q17	1820	1533	1511	1511
q18	8159	8055	7837	7837
q19	1883	1802	1542	1542
q20	2102	1940	1870	1870
q21	5259	4868	5098	4868
q22	663	577	623	577
Total cold run time: 73809 ms
Total hot run time: 56294 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173200 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 35210d8492e7e5e528c8d4346dde19d270998514, data reload: false

query1	2192	384	387	384
query2	7965	2432	2285	2285
query3	9684	207	213	207
query4	19021	17566	17290	17290
query5	3809	490	456	456
query6	247	195	153	153
query7	4587	293	281	281
query8	339	306	300	300
query9	8440	2366	2356	2356
query10	564	306	282	282
query11	10665	9847	9935	9847
query12	117	88	85	85
query13	1648	384	375	375
query14	10364	7070	7218	7070
query15	265	188	184	184
query16	8065	269	264	264
query17	1910	524	525	524
query18	2107	263	272	263
query19	309	153	159	153
query20	89	85	81	81
query21	209	131	126	126
query22	4338	3917	3923	3917
query23	34025	33492	33487	33487
query24	10202	2861	2967	2861
query25	592	390	368	368
query26	704	167	162	162
query27	2154	324	341	324
query28	6069	2159	2130	2130
query29	886	670	650	650
query30	244	164	166	164
query31	1001	775	743	743
query32	99	56	53	53
query33	657	285	283	283
query34	895	501	511	501
query35	765	629	643	629
query36	1176	998	998	998
query37	140	80	75	75
query38	2912	2874	2843	2843
query39	898	839	828	828
query40	208	135	127	127
query41	57	50	52	50
query42	109	100	99	99
query43	608	562	548	548
query44	1095	725	750	725
query45	201	164	162	162
query46	1087	702	710	702
query47	1821	1770	1738	1738
query48	374	304	309	304
query49	844	408	411	408
query50	776	385	380	380
query51	6924	6847	6706	6706
query52	111	96	94	94
query53	375	297	296	296
query54	868	449	445	445
query55	74	70	73	70
query56	301	267	267	267
query57	1148	1059	1068	1059
query58	257	255	251	251
query59	3468	3213	3336	3213
query60	299	274	280	274
query61	96	89	94	89
query62	600	440	420	420
query63	323	298	293	293
query64	8641	2212	1748	1748
query65	3150	3098	3097	3097
query66	756	315	326	315
query67	15499	14912	14742	14742
query68	5151	545	544	544
query69	654	440	322	322
query70	1171	1144	1109	1109
query71	447	277	279	277
query72	7238	5701	5372	5372
query73	784	331	321	321
query74	5901	5474	5450	5450
query75	3799	2609	2697	2609
query76	3523	1012	927	927
query77	607	295	295	295
query78	10355	10050	9686	9686
query79	2726	501	514	501
query80	1430	481	470	470
query81	553	216	226	216
query82	781	115	107	107
query83	190	169	168	168
query84	269	90	87	87
query85	1245	278	268	268
query86	455	291	293	291
query87	3286	3089	3077	3077
query88	4066	2441	2455	2441
query89	483	411	409	409
query90	1768	191	191	191
query91	126	99	98	98
query92	60	51	50	50
query93	3075	515	508	508
query94	1022	189	183	183
query95	406	316	317	316
query96	616	279	269	269
query97	3162	3022	3004	3004
query98	224	198	193	193
query99	1216	869	837	837
Total cold run time: 277212 ms
Total hot run time: 173200 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 35210d8492e7e5e528c8d4346dde19d270998514, data reload: false

query1	0.04	0.03	0.03
query2	0.09	0.04	0.03
query3	0.23	0.05	0.04
query4	1.67	0.07	0.07
query5	0.49	0.48	0.47
query6	1.14	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.55	0.49	0.49
query10	0.53	0.53	0.55
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.79	0.78	0.80
query15	0.86	0.82	0.81
query16	0.37	0.37	0.37
query17	1.05	1.00	0.98
query18	0.23	0.26	0.22
query19	1.93	1.70	1.81
query20	0.01	0.01	0.01
query21	15.46	0.77	0.66
query22	4.34	7.23	1.91
query23	18.29	1.37	1.23
query24	2.10	0.22	0.23
query25	0.16	0.09	0.09
query26	0.27	0.18	0.17
query27	0.08	0.07	0.08
query28	13.28	1.01	1.00
query29	12.62	3.26	3.23
query30	0.26	0.06	0.06
query31	2.86	0.39	0.38
query32	3.29	0.48	0.47
query33	2.85	2.93	2.92
query34	17.02	4.41	4.39
query35	4.43	4.49	4.50
query36	0.64	0.48	0.47
query37	0.18	0.16	0.15
query38	0.15	0.16	0.15
query39	0.05	0.03	0.04
query40	0.16	0.15	0.14
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.48 s

dataroaring
dataroaring approved these changes Jul 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit ba750b8 into apache:master Jul 8, 2024
25 of 27 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@liutang123 @dataroaring
[opt](meta) put normal partitions into recyclebin when replacing part…
dffb1ff 
…itions (#36361)

## Proposed changes

Issue Number: close #36714

Move the old partitions to recycle bin when replacing partition.

---------

Co-authored-by: liutang123 <liulijia@gmail.com>
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zclllyybb zclllyybb zclllyybb left review comments

@dataroaring dataroaring dataroaring approved these changes

@zddr zddr zddr approved these changes

@CalvinKirs CalvinKirs Awaiting requested review from CalvinKirs CalvinKirs is a code owner

@morningman morningman Awaiting requested review from morningman morningman is a code owner

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Enhancement] (nereids) Support fore keyword in replace partition statment
5 participants
@liutang123 @doris-robot @dataroaring @zddr @zclllyybb