Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](mtmv) Fix rewrite wrongly when sync and async materialized view name is same #37311

Merged
merged 3 commits into from
Jul 8, 2024

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 4, 2024

Proposed changes

This is brought by #33699

AbstractMaterializedViewRule#isMaterializationValid check would fail when sync and async materialized view name is same, because the Materialization valid check result is cached by name。this will cause the async materialized view check fail wrongly. This fix this by make sync materialized view name unique in every place.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

seawinde commented Jul 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39894 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1b57c25a422408f21bb8d0b22a1ede2d1a71e476, data reload: false

------ Round 1 ----------------------------------
q1	18077	4381	4376	4376
q2	2535	208	201	201
q3	11477	1187	1138	1138
q4	10846	745	786	745
q5	7513	2708	2691	2691
q6	224	142	139	139
q7	974	627	616	616
q8	9397	2065	2082	2065
q9	8882	6465	6452	6452
q10	8923	3770	3720	3720
q11	471	241	235	235
q12	474	239	234	234
q13	17771	3017	3024	3017
q14	274	227	229	227
q15	532	481	488	481
q16	504	390	376	376
q17	978	629	558	558
q18	8114	7437	7422	7422
q19	5998	1457	1358	1358
q20	666	341	329	329
q21	5035	3177	4020	3177
q22	419	337	346	337
Total cold run time: 120084 ms
Total hot run time: 39894 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4251	4208	4208
q2	364	262	263	262
q3	2964	2739	2779	2739
q4	2002	1594	1638	1594
q5	5261	5296	5267	5267
q6	222	130	134	130
q7	2071	1756	1774	1756
q8	3209	3324	3287	3287
q9	8363	8291	8324	8291
q10	3870	3663	3703	3663
q11	582	474	496	474
q12	796	622	626	622
q13	17425	2984	3017	2984
q14	288	278	254	254
q15	525	476	496	476
q16	465	438	437	437
q17	1784	1493	1472	1472
q18	7570	7448	7416	7416
q19	1711	1631	1605	1605
q20	1954	1795	1771	1771
q21	4881	4638	4726	4638
q22	625	542	550	542
Total cold run time: 71248 ms
Total hot run time: 53888 ms

@@ -72,7 +72,8 @@ Plan doGenerateScanPlan(CascadesContext cascadesContext) {
@Override
List<String> getMaterializationQualifier() {
return ImmutableList.of(olapTable.getDatabase().getCatalog().getName(),
ClusterNamespace.getNameFromFullName(olapTable.getDatabase().getFullName()), indexName);
ClusterNamespace.getNameFromFullName(olapTable.getDatabase().getFullName()),
olapTable.getName(), indexName);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add some cases?

@doris-robot
Copy link

TPC-DS: Total hot run time: 168493 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1b57c25a422408f21bb8d0b22a1ede2d1a71e476, data reload: false

query1	922	380	376	376
query2	6464	2506	2357	2357
query3	6645	202	206	202
query4	19575	17415	17133	17133
query5	4192	476	482	476
query6	262	180	167	167
query7	4609	309	301	301
query8	351	304	293	293
query9	8654	2399	2390	2390
query10	632	320	278	278
query11	10422	10138	9946	9946
query12	131	89	80	80
query13	1643	372	365	365
query14	10227	6817	6694	6694
query15	226	188	185	185
query16	7777	320	310	310
query17	1518	568	541	541
query18	1946	281	275	275
query19	196	150	155	150
query20	89	83	84	83
query21	214	134	132	132
query22	4344	3993	3991	3991
query23	33864	33193	33032	33032
query24	12337	2919	2833	2833
query25	689	379	389	379
query26	1796	155	153	153
query27	3044	314	321	314
query28	7372	2077	2074	2074
query29	1159	621	616	616
query30	283	151	152	151
query31	973	725	736	725
query32	92	56	57	56
query33	792	304	296	296
query34	951	462	489	462
query35	767	634	619	619
query36	1114	949	940	940
query37	283	80	80	80
query38	2866	2776	2745	2745
query39	863	788	797	788
query40	288	125	124	124
query41	56	55	53	53
query42	120	100	103	100
query43	566	523	559	523
query44	1171	729	739	729
query45	202	162	166	162
query46	1077	704	782	704
query47	1877	1743	1750	1743
query48	364	307	304	304
query49	1203	431	418	418
query50	769	394	397	394
query51	6937	6868	6738	6738
query52	103	95	94	94
query53	371	291	301	291
query54	1022	454	446	446
query55	78	77	76	76
query56	301	276	293	276
query57	1166	1047	1056	1047
query58	271	360	253	253
query59	3433	3360	3181	3181
query60	315	280	285	280
query61	98	94	93	93
query62	661	434	438	434
query63	323	304	290	290
query64	9881	2267	1749	1749
query65	3300	3114	3086	3086
query66	1380	364	326	326
query67	15800	14777	14832	14777
query68	8891	543	572	543
query69	729	492	330	330
query70	1268	1138	1077	1077
query71	547	276	269	269
query72	8479	4341	2611	2611
query73	1836	319	320	319
query74	5953	5496	5529	5496
query75	5144	2636	2665	2636
query76	5162	1044	923	923
query77	755	303	299	299
query78	9492	8833	8867	8833
query79	8513	516	545	516
query80	1006	463	473	463
query81	545	218	221	218
query82	454	102	107	102
query83	341	163	162	162
query84	273	87	82	82
query85	1093	272	265	265
query86	363	329	295	295
query87	3260	3077	3152	3077
query88	4031	2360	2352	2352
query89	525	373	367	367
query90	2221	182	183	182
query91	126	99	99	99
query92	66	47	95	47
query93	7045	496	497	496
query94	1448	213	203	203
query95	410	307	304	304
query96	601	266	268	266
query97	3130	3003	2960	2960
query98	223	205	189	189
query99	1176	852	833	833
Total cold run time: 298523 ms
Total hot run time: 168493 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.3 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1b57c25a422408f21bb8d0b22a1ede2d1a71e476, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.08	0.08
query5	0.51	0.49	0.48
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.54	0.50	0.50
query10	0.55	0.54	0.55
query11	0.15	0.11	0.11
query12	0.14	0.12	0.12
query13	0.58	0.58	0.59
query14	0.79	0.78	0.76
query15	0.84	0.81	0.82
query16	0.37	0.34	0.36
query17	0.96	1.02	0.96
query18	0.25	0.24	0.26
query19	1.90	1.75	1.73
query20	0.02	0.01	0.01
query21	15.41	0.76	0.66
query22	3.86	6.83	2.50
query23	18.28	1.36	1.32
query24	2.18	0.22	0.22
query25	0.15	0.09	0.09
query26	0.30	0.22	0.22
query27	0.46	0.24	0.23
query28	13.31	1.02	1.01
query29	12.67	3.34	3.28
query30	0.25	0.07	0.06
query31	2.88	0.40	0.40
query32	3.23	0.47	0.47
query33	2.87	2.97	2.95
query34	17.07	4.33	4.33
query35	4.43	4.40	4.42
query36	0.65	0.45	0.49
query37	0.20	0.14	0.14
query38	0.16	0.14	0.14
query39	0.05	0.03	0.04
query40	0.14	0.12	0.11
query41	0.09	0.04	0.04
query42	0.05	0.04	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.55 s
Total hot run time: 31.3 s

@seawinde
Copy link
Contributor Author

seawinde commented Jul 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39844 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 79e760b4608cd10237a8ebfdc50ca9387da7c6fb, data reload: false

------ Round 1 ----------------------------------
q1	17919	4493	4353	4353
q2	2503	197	186	186
q3	11496	1175	1100	1100
q4	10768	767	756	756
q5	7822	2648	2697	2648
q6	222	136	136	136
q7	959	600	609	600
q8	9564	2074	2049	2049
q9	8830	6498	6474	6474
q10	8947	3695	3680	3680
q11	458	239	239	239
q12	403	237	223	223
q13	17764	2973	2980	2973
q14	269	225	233	225
q15	532	491	489	489
q16	497	376	375	375
q17	958	769	668	668
q18	8037	7503	7326	7326
q19	3447	1450	1500	1450
q20	707	329	321	321
q21	5042	3231	3973	3231
q22	398	342	346	342
Total cold run time: 117542 ms
Total hot run time: 39844 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4339	4272	4242	4242
q2	376	266	264	264
q3	2957	2780	2754	2754
q4	1895	1610	1599	1599
q5	5259	5286	5252	5252
q6	214	128	129	128
q7	2049	1765	1777	1765
q8	3202	3292	3307	3292
q9	8304	8315	8347	8315
q10	3860	3617	3598	3598
q11	582	486	478	478
q12	774	584	630	584
q13	17355	2941	3010	2941
q14	297	260	260	260
q15	522	484	492	484
q16	457	411	428	411
q17	1771	1466	1485	1466
q18	7578	7525	7297	7297
q19	1704	1558	1643	1558
q20	1980	1793	1790	1790
q21	4780	4748	4835	4748
q22	642	538	565	538
Total cold run time: 70897 ms
Total hot run time: 53764 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171834 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 79e760b4608cd10237a8ebfdc50ca9387da7c6fb, data reload: false

query1	910	366	363	363
query2	7188	2291	2319	2291
query3	6648	209	220	209
query4	27886	17656	17238	17238
query5	4185	486	483	483
query6	309	172	165	165
query7	4605	293	300	293
query8	325	284	286	284
query9	8576	2436	2404	2404
query10	656	304	284	284
query11	11360	9967	10036	9967
query12	133	86	83	83
query13	1683	368	363	363
query14	10094	7537	7556	7537
query15	228	184	178	178
query16	7800	297	301	297
query17	1830	576	513	513
query18	1810	273	273	273
query19	203	155	145	145
query20	88	78	81	78
query21	213	149	124	124
query22	4362	4063	4065	4063
query23	33759	33126	33142	33126
query24	12022	2806	2845	2806
query25	669	357	377	357
query26	1753	156	155	155
query27	2930	317	316	316
query28	7597	2073	2065	2065
query29	1062	631	619	619
query30	286	145	146	145
query31	925	738	733	733
query32	106	52	56	52
query33	769	286	289	286
query34	965	476	476	476
query35	734	625	639	625
query36	1090	943	951	943
query37	217	82	71	71
query38	2904	2751	2730	2730
query39	871	784	813	784
query40	287	124	123	123
query41	52	52	49	49
query42	119	94	106	94
query43	606	514	544	514
query44	1217	745	735	735
query45	186	162	161	161
query46	1084	709	730	709
query47	1849	1780	1795	1780
query48	387	294	296	294
query49	1193	412	410	410
query50	780	382	395	382
query51	6786	6785	6728	6728
query52	112	90	93	90
query53	359	295	288	288
query54	956	448	442	442
query55	76	73	74	73
query56	323	270	267	267
query57	1175	1068	1048	1048
query58	252	246	265	246
query59	3279	3140	3230	3140
query60	311	276	276	276
query61	95	116	91	91
query62	637	452	456	452
query63	319	298	292	292
query64	10410	2167	1672	1672
query65	3217	3076	3095	3076
query66	1339	339	339	339
query67	15266	14907	15232	14907
query68	5562	544	531	531
query69	682	411	356	356
query70	1115	1112	1111	1111
query71	442	277	269	269
query72	7295	4985	5432	4985
query73	772	323	316	316
query74	5874	5468	5403	5403
query75	3711	2687	2689	2687
query76	3438	927	926	926
query77	636	326	310	310
query78	9367	8949	8732	8732
query79	1782	522	519	519
query80	2025	474	460	460
query81	573	223	221	221
query82	768	107	109	107
query83	257	163	166	163
query84	270	91	87	87
query85	1442	377	302	302
query86	471	326	303	303
query87	3318	3065	3128	3065
query88	4005	2368	2351	2351
query89	477	385	380	380
query90	1828	190	192	190
query91	131	100	100	100
query92	59	49	51	49
query93	1857	513	512	512
query94	1194	212	208	208
query95	403	316	315	315
query96	598	276	265	265
query97	3249	2995	2985	2985
query98	233	197	196	196
query99	1182	828	870	828
Total cold run time: 287439 ms
Total hot run time: 171834 ms

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@doris-robot
Copy link

ClickBench: Total hot run time: 31.24 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 79e760b4608cd10237a8ebfdc50ca9387da7c6fb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.50
query6	1.14	0.73	0.72
query7	0.02	0.02	0.02
query8	0.06	0.05	0.05
query9	0.54	0.49	0.50
query10	0.56	0.54	0.55
query11	0.15	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.57
query14	0.77	0.78	0.78
query15	0.85	0.80	0.80
query16	0.36	0.36	0.36
query17	0.96	0.98	0.96
query18	0.20	0.24	0.27
query19	1.82	1.86	1.71
query20	0.01	0.01	0.01
query21	15.40	0.75	0.66
query22	4.07	5.89	2.54
query23	18.28	1.35	1.25
query24	2.15	0.22	0.22
query25	0.15	0.11	0.08
query26	0.30	0.21	0.21
query27	0.45	0.23	0.24
query28	13.23	1.02	1.00
query29	12.58	3.34	3.32
query30	0.26	0.06	0.06
query31	2.88	0.40	0.39
query32	3.26	0.50	0.47
query33	2.91	2.87	2.90
query34	17.15	4.36	4.39
query35	4.42	4.42	4.38
query36	0.63	0.46	0.48
query37	0.19	0.16	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.16	0.13	0.12
query41	0.09	0.06	0.04
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.54 s
Total hot run time: 31.24 s

@starocean999 starocean999 merged commit 5c03890 into apache:master Jul 8, 2024
26 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
… name is same (#37311)

## Proposed changes

This is brought by #33699

`AbstractMaterializedViewRule#isMaterializationValid` check would fail
when sync and async materialized view name is same, because the
Materialization valid check result is cached by name。this will cause the
async materialized view check fail wrongly. This fix this by make sync
materialized view name unique in every place.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged not-merge/2.1 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants