Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhance](mtmv)when calculating the availability of MTMV, no longer consider refresh state #36507

Merged
merged 3 commits into from
Jun 20, 2024
Merged

[enhance](mtmv)when calculating the availability of MTMV, no longer consider refresh state #36507

merged 3 commits into from
Jun 20, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Jun 19, 2024

Before the modification, if the last refresh of the materialized view failed, the entire materialized view cannot be used for transparent rewriting. In fact, many partitions may be available, so this modification was made

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Jun 19, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40305 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3395af5f5cfb783f080914040f72c80325ffa70f, data reload: false

------ Round 1 ----------------------------------
q1	18130	4841	4404	4404
q2	2608	194	210	194
q3	11347	1135	1160	1135
q4	10758	822	812	812
q5	7818	2759	2701	2701
q6	236	142	144	142
q7	972	643	629	629
q8	9306	2135	2177	2135
q9	9149	6537	6541	6537
q10	8919	3743	3738	3738
q11	464	241	244	241
q12	442	243	232	232
q13	18832	2967	3002	2967
q14	260	230	223	223
q15	534	473	473	473
q16	515	399	378	378
q17	996	674	776	674
q18	8115	7483	7360	7360
q19	2715	1457	1391	1391
q20	665	307	327	307
q21	5004	3284	4042	3284
q22	418	348	350	348
Total cold run time: 118203 ms
Total hot run time: 40305 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4354	4303	4306	4303
q2	367	265	255	255
q3	2982	2789	2702	2702
q4	1960	1634	1613	1613
q5	5269	5292	5263	5263
q6	219	127	132	127
q7	2150	1732	1735	1732
q8	3227	3350	3327	3327
q9	8426	8354	8401	8354
q10	3896	3710	3699	3699
q11	600	496	491	491
q12	759	594	578	578
q13	16429	3029	3020	3020
q14	296	256	246	246
q15	526	483	484	483
q16	456	418	422	418
q17	1784	1489	1475	1475
q18	7725	7527	7330	7330
q19	1706	1459	1469	1459
q20	1989	1781	1757	1757
q21	4798	4751	4714	4714
q22	627	542	544	542
Total cold run time: 70545 ms
Total hot run time: 53888 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172537 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3395af5f5cfb783f080914040f72c80325ffa70f, data reload: false

query1	929	377	380	377
query2	6477	2445	2303	2303
query3	6648	205	207	205
query4	21422	17268	17361	17268
query5	4148	470	468	468
query6	256	171	166	166
query7	4598	296	290	290
query8	309	291	286	286
query9	8649	2383	2377	2377
query10	630	347	288	288
query11	10498	10048	10045	10045
query12	137	87	83	83
query13	1671	365	372	365
query14	10140	6874	7709	6874
query15	243	189	190	189
query16	7843	265	266	265
query17	1905	528	541	528
query18	1953	272	272	272
query19	192	153	156	153
query20	94	85	80	80
query21	209	134	133	133
query22	4589	3925	4117	3925
query23	33649	33114	32946	32946
query24	12058	2744	2840	2744
query25	657	361	359	359
query26	1797	157	154	154
query27	3074	317	318	317
query28	7673	2042	2034	2034
query29	1127	621	615	615
query30	281	153	147	147
query31	936	738	751	738
query32	89	54	56	54
query33	767	293	310	293
query34	977	466	469	466
query35	736	617	615	615
query36	1065	969	951	951
query37	284	72	72	72
query38	2869	2765	2728	2728
query39	851	801	782	782
query40	280	129	129	129
query41	54	50	55	50
query42	121	97	101	97
query43	616	562	553	553
query44	1203	720	729	720
query45	200	167	177	167
query46	1085	754	744	744
query47	1824	1748	1736	1736
query48	374	293	301	293
query49	1205	403	406	403
query50	771	391	382	382
query51	6782	6710	6694	6694
query52	102	92	96	92
query53	358	289	302	289
query54	970	440	442	440
query55	77	72	74	72
query56	276	261	261	261
query57	1117	1058	1039	1039
query58	256	248	258	248
query59	3386	3310	3218	3218
query60	289	268	270	268
query61	102	91	91	91
query62	654	472	456	456
query63	319	297	294	294
query64	9912	2262	1745	1745
query65	3183	3113	3092	3092
query66	1372	339	324	324
query67	15537	14976	14925	14925
query68	4689	553	546	546
query69	572	439	396	396
query70	1159	1162	1176	1162
query71	431	271	273	271
query72	7468	5519	5279	5279
query73	750	325	319	319
query74	5880	5476	5483	5476
query75	3461	2653	2673	2653
query76	2961	958	968	958
query77	634	298	300	298
query78	10367	9999	9817	9817
query79	1156	527	524	524
query80	691	477	457	457
query81	575	225	217	217
query82	610	102	105	102
query83	189	169	168	168
query84	244	85	83	83
query85	1255	286	269	269
query86	432	314	299	299
query87	3247	3056	3071	3056
query88	3933	2335	2332	2332
query89	483	410	389	389
query90	1817	195	199	195
query91	145	174	98	98
query92	65	51	53	51
query93	1912	529	511	511
query94	1177	198	188	188
query95	410	308	314	308
query96	597	269	262	262
query97	3303	3070	3040	3040
query98	212	198	198	198
query99	1216	838	841	838
Total cold run time: 276896 ms
Total hot run time: 172537 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3395af5f5cfb783f080914040f72c80325ffa70f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.22	0.04	0.04
query4	1.68	0.07	0.07
query5	0.48	0.49	0.51
query6	1.13	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.49	0.48
query10	0.54	0.54	0.54
query11	0.14	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.60	0.61
query14	0.78	0.79	0.76
query15	0.84	0.81	0.81
query16	0.37	0.36	0.35
query17	1.00	1.06	1.03
query18	0.22	0.26	0.25
query19	1.77	1.74	1.83
query20	0.01	0.01	0.01
query21	15.41	0.67	0.67
query22	4.83	7.66	1.61
query23	18.32	1.37	1.22
query24	2.14	0.23	0.22
query25	0.15	0.08	0.08
query26	0.26	0.19	0.18
query27	0.08	0.08	0.07
query28	13.19	1.02	1.00
query29	12.65	3.30	3.27
query30	0.27	0.07	0.05
query31	2.88	0.39	0.39
query32	3.25	0.49	0.47
query33	2.87	2.94	2.94
query34	16.93	4.42	4.39
query35	4.54	4.45	4.48
query36	0.64	0.47	0.47
query37	0.18	0.16	0.16
query38	0.15	0.15	0.14
query39	0.05	0.03	0.03
query40	0.18	0.15	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.81 s
Total hot run time: 30.32 s

@zddr
Copy link
Contributor Author

zddr commented Jun 19, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40068 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 80f3a4b710791adb22989db1849fa837a2825ddb, data reload: false

------ Round 1 ----------------------------------
q1	18002	4633	4380	4380
q2	2625	193	201	193
q3	11668	1116	1165	1116
q4	10363	837	873	837
q5	7503	2704	2699	2699
q6	225	167	137	137
q7	975	610	613	610
q8	9426	2097	2056	2056
q9	8624	6434	6446	6434
q10	8917	3745	3739	3739
q11	440	236	238	236
q12	394	238	225	225
q13	17767	2984	3015	2984
q14	263	211	217	211
q15	519	481	476	476
q16	513	387	375	375
q17	955	698	659	659
q18	8024	7346	7342	7342
q19	6707	1479	1517	1479
q20	646	300	330	300
q21	4928	3252	3970	3252
q22	405	336	328	328
Total cold run time: 119889 ms
Total hot run time: 40068 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4422	4235	4208	4208
q2	373	265	260	260
q3	2982	2747	2681	2681
q4	1846	1586	1583	1583
q5	5229	5269	5232	5232
q6	220	126	129	126
q7	2119	1700	1660	1660
q8	3181	3310	3317	3310
q9	8258	8290	8234	8234
q10	3899	3652	3649	3649
q11	580	493	476	476
q12	761	606	580	580
q13	17450	2986	2977	2977
q14	299	261	253	253
q15	520	465	473	465
q16	459	398	422	398
q17	1767	1469	1456	1456
q18	7717	7445	7262	7262
q19	1673	1514	1472	1472
q20	1965	1769	1753	1753
q21	4742	4662	4692	4662
q22	627	556	561	556
Total cold run time: 71089 ms
Total hot run time: 53253 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170933 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 80f3a4b710791adb22989db1849fa837a2825ddb, data reload: false

query1	927	378	376	376
query2	6447	2429	2274	2274
query3	6645	206	208	206
query4	19608	17233	17395	17233
query5	4634	510	475	475
query6	264	171	163	163
query7	4897	308	292	292
query8	338	316	306	306
query9	8721	2374	2377	2374
query10	609	310	297	297
query11	10561	9994	10087	9994
query12	136	84	85	84
query13	1643	362	361	361
query14	9478	6918	6131	6131
query15	230	185	187	185
query16	7836	273	259	259
query17	1692	533	536	533
query18	1899	276	273	273
query19	190	155	155	155
query20	92	83	78	78
query21	202	131	133	131
query22	4243	3957	3893	3893
query23	33938	33028	33078	33028
query24	11537	2751	2902	2751
query25	623	350	357	350
query26	1689	152	150	150
query27	3105	312	319	312
query28	7229	2014	2018	2014
query29	1034	643	590	590
query30	288	148	148	148
query31	944	715	752	715
query32	97	57	56	56
query33	765	284	282	282
query34	972	486	453	453
query35	743	611	607	607
query36	1126	926	885	885
query37	158	72	68	68
query38	2856	2744	2746	2744
query39	856	791	793	791
query40	289	128	131	128
query41	59	53	52	52
query42	121	99	104	99
query43	579	551	544	544
query44	1233	716	731	716
query45	195	167	170	167
query46	1095	714	712	712
query47	1875	1752	1756	1752
query48	376	303	294	294
query49	1183	407	400	400
query50	753	381	389	381
query51	6784	6698	6634	6634
query52	106	95	94	94
query53	353	291	289	289
query54	998	436	434	434
query55	74	73	74	73
query56	283	254	254	254
query57	1145	1021	1027	1021
query58	250	236	245	236
query59	3459	3234	2979	2979
query60	289	279	272	272
query61	93	92	89	89
query62	641	484	450	450
query63	321	282	285	282
query64	9914	2246	1723	1723
query65	3205	3138	3127	3127
query66	1358	333	327	327
query67	15566	14968	14894	14894
query68	5697	540	560	540
query69	608	449	421	421
query70	1159	1150	1039	1039
query71	446	277	278	277
query72	7712	5475	4975	4975
query73	758	324	325	324
query74	5875	5580	5562	5562
query75	3742	2658	2656	2656
query76	3263	930	932	930
query77	668	304	298	298
query78	10407	9802	9832	9802
query79	2998	508	517	508
query80	1181	480	495	480
query81	535	223	221	221
query82	586	111	102	102
query83	203	175	174	174
query84	276	91	85	85
query85	1398	344	325	325
query86	410	321	311	311
query87	3269	3049	3115	3049
query88	3628	2354	2336	2336
query89	487	381	407	381
query90	1874	199	198	198
query91	141	109	113	109
query92	64	51	52	51
query93	4478	528	508	508
query94	1198	197	198	197
query95	417	321	323	321
query96	595	261	265	261
query97	3362	3038	3015	3015
query98	212	192	191	191
query99	1305	888	846	846
Total cold run time: 280801 ms
Total hot run time: 170933 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.61 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 80f3a4b710791adb22989db1849fa837a2825ddb, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.04	0.05
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.49
query6	1.12	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.54	0.53	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.76	0.77	0.78
query15	0.85	0.80	0.80
query16	0.37	0.36	0.35
query17	0.95	0.97	1.03
query18	0.21	0.24	0.25
query19	1.86	1.70	1.69
query20	0.01	0.02	0.01
query21	15.43	0.63	0.64
query22	3.43	7.64	2.11
query23	18.29	1.30	1.34
query24	2.06	0.24	0.22
query25	0.16	0.08	0.09
query26	0.27	0.19	0.18
query27	0.07	0.08	0.08
query28	13.20	1.01	1.00
query29	12.65	3.33	3.25
query30	0.25	0.08	0.05
query31	2.84	0.39	0.38
query32	3.27	0.48	0.46
query33	2.84	2.91	2.90
query34	16.95	4.40	4.38
query35	4.47	4.47	4.46
query36	0.66	0.46	0.45
query37	0.18	0.14	0.14
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.19	0.14	0.14
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.3 s
Total hot run time: 30.61 s

@zddr
Copy link
Contributor Author

zddr commented Jun 20, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 20, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40823 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f94bfedc9a3603e524da67c822fcc266fb5acfa9, data reload: false

------ Round 1 ----------------------------------
q1	17614	4280	4250	4250
q2	2035	190	196	190
q3	10439	1231	1090	1090
q4	10197	830	734	734
q5	7461	2633	2617	2617
q6	224	138	133	133
q7	946	609	588	588
q8	9216	2049	2071	2049
q9	9030	6490	6445	6445
q10	8916	3747	3715	3715
q11	469	239	238	238
q12	496	236	232	232
q13	17767	2980	2983	2980
q14	264	222	229	222
q15	519	484	472	472
q16	537	381	389	381
q17	968	626	678	626
q18	8102	7989	7822	7822
q19	8963	1455	1492	1455
q20	708	331	315	315
q21	5102	3916	4163	3916
q22	412	361	353	353
Total cold run time: 120385 ms
Total hot run time: 40823 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4624	4432	4344	4344
q2	367	266	254	254
q3	3083	2901	2949	2901
q4	2005	1684	1710	1684
q5	5476	5516	5473	5473
q6	217	127	129	127
q7	2203	1901	1835	1835
q8	3277	3437	3386	3386
q9	8722	8683	8706	8683
q10	4062	3837	3866	3837
q11	574	504	535	504
q12	831	593	603	593
q13	16719	3073	3127	3073
q14	309	263	262	262
q15	526	474	471	471
q16	482	433	434	433
q17	1787	1478	1480	1478
q18	8014	7754	7735	7735
q19	1792	1642	1580	1580
q20	2139	1840	1838	1838
q21	5019	4715	5004	4715
q22	660	534	561	534
Total cold run time: 72888 ms
Total hot run time: 55740 ms

@morrySnow morrySnow merged commit 8393181 into apache:master Jun 20, 2024
27 of 29 checks passed
zddr added a commit to zddr/incubator-doris that referenced this pull request Jun 20, 2024
@zddr
[enhance](mtmv)when calculating the availability of MTMV, no longer c…
ad75d30 
…onsider refresh state (apache#36507)

Before the modification, if the last refresh of the materialized view
failed, the entire materialized view cannot be used for transparent
rewriting. In fact, many partitions may be available, so this
modification was made
@zddr zddr mentioned this pull request Jun 20, 2024
Merged
iszhangpch pushed a commit to iszhangpch/doris-p that referenced this pull request Jun 21, 2024
@zddr
[enhance](mtmv)when calculating the availability of MTMV, no longer c…
a976889 
…onsider refresh state (apache#36507)

Before the modification, if the last refresh of the materialized view
failed, the entire materialized view cannot be used for transparent
rewriting. In fact, many partitions may be available, so this
modification was made
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
@zddr @dataroaring
[enhance](mtmv)when calculating the availability of MTMV, no longer c…
13010a5 
…onsider refresh state (#36507)

Before the modification, if the last refresh of the materialized view
failed, the entire materialized view cannot be used for transparent
rewriting. In fact, many partitions may be available, so this
modification was made
morningman pushed a commit that referenced this pull request Jul 2, 2024
@zddr
[enhance](mtmv)when calculating the availability of MTMV, no longer c… (
ebc1afe 
#36617)

pick: #36507
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seawinde seawinde seawinde approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.0-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @doris-robot @morrySnow @seawinde @morningman @dataroaring