Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](mtmv)fix second level MTMV will refresh all partition #38698

Merged
merged 5 commits into from
Aug 2, 2024
Merged

[fix](mtmv)fix second level MTMV will refresh all partition #38698

merged 5 commits into from
Aug 2, 2024

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Aug 1, 2024

if:

  • create mv2 as select * from mv1;
  • create mv1 as select * from t1;
  • t1 has 2 partitions: p1,p2

when t1 insert data to p1;

mv1 will refresh p1

mv2 will refresh p1 and p2

fix:
should not check if data is sync between mv2 and t1

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@zddr
Copy link
Contributor Author

zddr commented Aug 1, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 41909 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 740587d0a501275af16ec2469d2d2266426734df, data reload: false

------ Round 1 ----------------------------------
q1	17599	4204	4079	4079
q2	2032	202	235	202
q3	10414	1315	1398	1315
q4	10164	909	927	909
q5	7645	3052	2984	2984
q6	230	138	139	138
q7	1048	627	615	615
q8	9441	1938	1952	1938
q9	10248	6592	6619	6592
q10	8783	3821	3819	3819
q11	428	249	246	246
q12	455	231	230	230
q13	17762	2994	2951	2951
q14	276	247	243	243
q15	550	479	486	479
q16	519	388	391	388
q17	978	945	907	907
q18	8244	7265	7341	7265
q19	1373	1216	1225	1216
q20	573	314	337	314
q21	5339	4799	4801	4799
q22	352	280	286	280
Total cold run time: 114453 ms
Total hot run time: 41909 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4029	3998	3985	3985
q2	330	225	217	217
q3	2985	3016	3029	3016
q4	1907	1911	1886	1886
q5	5256	5210	5289	5210
q6	218	132	127	127
q7	2076	1685	1727	1685
q8	3212	3292	3307	3292
q9	8323	8325	8258	8258
q10	3766	3873	3874	3873
q11	540	464	453	453
q12	740	557	577	557
q13	10091	2967	2985	2967
q14	282	263	255	255
q15	512	478	473	473
q16	466	404	395	395
q17	1740	1699	1677	1677
q18	7734	7408	7187	7187
q19	2779	1668	1664	1664
q20	1970	1769	1737	1737
q21	5516	5317	5202	5202
q22	510	456	439	439
Total cold run time: 64982 ms
Total hot run time: 54555 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 168553 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 740587d0a501275af16ec2469d2d2266426734df, data reload: false

query1	916	373	369	369
query2	6507	1778	1714	1714
query3	6675	216	226	216
query4	20359	17317	17309	17309
query5	4278	526	518	518
query6	305	179	172	172
query7	4623	306	291	291
query8	251	222	196	196
query9	8511	2363	2353	2353
query10	423	280	264	264
query11	10666	10030	9948	9948
query12	135	88	87	87
query13	1627	369	369	369
query14	9527	6942	7521	6942
query15	210	158	159	158
query16	7119	459	438	438
query17	967	573	563	563
query18	1897	280	278	278
query19	194	146	142	142
query20	92	86	84	84
query21	208	101	101	101
query22	4224	4094	3979	3979
query23	33893	32755	32924	32755
query24	10259	3046	3074	3046
query25	720	384	391	384
query26	1752	155	158	155
query27	2869	276	280	276
query28	6842	1960	1958	1958
query29	1306	422	420	420
query30	288	151	156	151
query31	947	752	761	752
query32	104	58	55	55
query33	709	307	320	307
query34	918	487	484	484
query35	857	712	747	712
query36	1016	865	883	865
query37	221	78	81	78
query38	2848	2796	2802	2796
query39	878	818	817	817
query40	282	116	111	111
query41	51	44	41	41
query42	119	100	110	100
query43	486	440	455	440
query44	1208	727	719	719
query45	207	176	177	176
query46	1097	838	796	796
query47	1796	1680	1716	1680
query48	379	293	291	291
query49	1181	416	429	416
query50	910	444	434	434
query51	6802	6771	6629	6629
query52	108	92	91	91
query53	263	190	190	190
query54	657	448	444	444
query55	79	75	79	75
query56	281	258	259	258
query57	1152	1033	1003	1003
query58	312	256	270	256
query59	2732	2561	2343	2343
query60	291	268	278	268
query61	99	92	95	92
query62	928	646	672	646
query63	226	194	185	185
query64	5880	1932	1938	1932
query65	3140	3069	3087	3069
query66	1432	345	342	342
query67	15354	14851	14891	14851
query68	4422	570	596	570
query69	650	368	325	325
query70	1178	1070	1078	1070
query71	448	290	293	290
query72	7179	2695	2483	2483
query73	818	328	338	328
query74	6001	5639	5658	5639
query75	3607	2741	2730	2730
query76	3068	1242	1280	1242
query77	611	305	314	305
query78	9452	8910	8866	8866
query79	2792	534	546	534
query80	963	522	508	508
query81	557	227	222	222
query82	963	132	130	130
query83	323	178	172	172
query84	275	79	79	79
query85	1321	335	306	306
query86	468	300	297	297
query87	3263	3064	3114	3064
query88	3960	2401	2411	2401
query89	396	292	290	290
query90	1827	205	199	199
query91	128	106	103	103
query92	71	51	54	51
query93	2109	615	616	615
query94	893	295	309	295
query95	372	276	276	276
query96	625	285	287	285
query97	3241	3024	3068	3024
query98	232	202	192	192
query99	1689	1285	1282	1282
Total cold run time: 267495 ms
Total hot run time: 168553 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 740587d0a501275af16ec2469d2d2266426734df, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.06
query4	1.65	0.09	0.07
query5	0.49	0.49	0.49
query6	1.14	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.58	0.50	0.50
query10	0.56	0.55	0.55
query11	0.16	0.12	0.12
query12	0.14	0.13	0.12
query13	0.61	0.61	0.60
query14	0.78	0.79	0.80
query15	0.89	0.86	0.86
query16	0.36	0.35	0.35
query17	1.01	0.99	1.00
query18	0.22	0.25	0.24
query19	1.84	1.75	1.70
query20	0.01	0.01	0.01
query21	15.40	0.78	0.66
query22	4.28	7.79	1.24
query23	17.83	1.46	1.40
query24	2.32	0.21	0.22
query25	0.18	0.08	0.08
query26	0.32	0.22	0.21
query27	0.46	0.23	0.23
query28	13.17	1.00	0.96
query29	12.57	3.34	3.29
query30	0.26	0.05	0.06
query31	2.84	0.40	0.40
query32	3.23	0.50	0.48
query33	2.97	3.02	2.97
query34	15.43	4.30	4.26
query35	4.29	4.29	4.29
query36	0.68	0.47	0.47
query37	0.19	0.17	0.16
query38	0.16	0.16	0.14
query39	0.04	0.04	0.04
query40	0.15	0.12	0.13
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 107.83 s
Total hot run time: 30.06 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 1, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 1, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow removed the doing label Aug 1, 2024
@morrySnow morrySnow added the p0_b label Aug 2, 2024
@morrySnow morrySnow merged commit 023fa8d into apache:master Aug 2, 2024
30 of 31 checks passed
dataroaring pushed a commit that referenced this pull request Aug 2, 2024
@zddr @dataroaring
[fix](mtmv) second level MTMV always refresh all partition by mistake (…
7ec1a8e 
…#38698)

if:
-  create mv2 as select * from mv1;
- create mv1 as select * from t1;
- t1 has 2 partitions: p1,p2

when t1 insert data to p1;

mv1 will refresh p1

mv2 will refresh p1 and p2

fix:
should not check if data is sync between mv2 and t1
zddr added a commit to zddr/incubator-doris that referenced this pull request Aug 9, 2024
@zddr
[fix](mtmv) second level MTMV always refresh all partition by mistake (…
7347517 
…apache#38698)

if:
-  create mv2 as select * from mv1;
- create mv1 as select * from t1;
- t1 has 2 partitions: p1,p2

when t1 insert data to p1;

mv1 will refresh p1

mv2 will refresh p1 and p2

fix:
should not check if data is sync between mv2 and t1
@zddr zddr mentioned this pull request Aug 9, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 10, 2024
@zddr
[fix](mtmv) second level MTMV always refresh all partition by mistake… (
9b4354f 
#39167)

pick: 
#38698
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@seawinde seawinde seawinde approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged p0_b reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zddr @doris-robot @morrySnow @seawinde @yiguolei @dataroaring