[fix](mtmv) Fix mv rewrite fail when base table add column #44867

seawinde · 2024-12-02T09:43:51Z

What problem does this PR solve?

Fix mv rewrite fail when base table add column
Such as mv def is as following:

            select
            o_orderstatus,
            l_suppkey,
            sum_merge(sum_state(o_shippriority)),
            group_concat_merge(group_concat_state(o_orderstatus)),
            avg_merge(avg_state(l_linenumber)),
            max_by_merge(max_by_state(O_COMMENT,o_totalprice)),
            count_merge(count_state(l_orderkey)),
            multi_distinct_count_merge(multi_distinct_count_state(l_shipmode))
            from lineitem
            left join orders 
            on l_orderkey = o_orderkey and l_shipdate = o_orderdate
            group by
            o_orderstatus,
            l_suppkey
            order by o_orderstatus;

if add column to table orders,
ALTER TABLE orders ADD COLUMN new_col INT KEY DEFAULT "0"; would cause rewrite fail
should rewrite successfully, this pr fix this

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

Fix mv rewrite fail when base table add column

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

doris-robot · 2024-12-02T09:43:57Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

seawinde · 2024-12-02T09:44:22Z

run buildall

doris-robot · 2024-12-02T10:23:18Z

TPC-H: Total hot run time: 40095 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 817f076163b0dbf1003b3eb7240e530147b68635, data reload: false

------ Round 1 ----------------------------------
q1	17621	7406	7245	7245
q2	2040	179	169	169
q3	10551	1077	1224	1077
q4	10442	809	750	750
q5	7602	2736	2735	2735
q6	237	147	149	147
q7	992	641	592	592
q8	9248	1831	1924	1831
q9	6676	6429	6531	6429
q10	7044	2311	2308	2308
q11	461	258	264	258
q12	429	223	220	220
q13	17803	3039	3036	3036
q14	247	223	211	211
q15	577	539	503	503
q16	675	616	589	589
q17	964	605	573	573
q18	7242	6747	6719	6719
q19	1337	996	1017	996
q20	456	178	183	178
q21	4010	3211	3214	3211
q22	378	323	318	318
Total cold run time: 107032 ms
Total hot run time: 40095 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7216	7202	7229	7202
q2	322	235	236	235
q3	2888	2818	2925	2818
q4	2114	1863	1910	1863
q5	5673	5683	5666	5666
q6	235	145	149	145
q7	2230	1861	1819	1819
q8	3430	3523	3472	3472
q9	9005	9004	9019	9004
q10	3634	3594	3541	3541
q11	593	509	502	502
q12	816	609	595	595
q13	10539	3278	3265	3265
q14	302	290	288	288
q15	575	519	539	519
q16	681	652	660	652
q17	1839	1618	1593	1593
q18	8385	7683	7658	7658
q19	1691	1594	1558	1558
q20	2144	1876	1880	1876
q21	5754	5624	5554	5554
q22	674	572	584	572
Total cold run time: 70740 ms
Total hot run time: 60397 ms

doris-robot · 2024-12-02T10:35:59Z

TPC-DS: Total hot run time: 198802 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 817f076163b0dbf1003b3eb7240e530147b68635, data reload: false

query1	1250	947	925	925
query2	6275	2075	2039	2039
query3	11112	4605	4561	4561
query4	66623	28300	23715	23715
query5	4907	472	457	457
query6	406	194	178	178
query7	5469	316	301	301
query8	328	244	246	244
query9	8672	2704	2700	2700
query10	439	277	246	246
query11	17272	15477	15847	15477
query12	152	103	113	103
query13	1448	405	429	405
query14	10956	8084	7439	7439
query15	215	183	205	183
query16	7162	488	472	472
query17	1025	574	556	556
query18	2053	319	305	305
query19	197	154	149	149
query20	115	109	116	109
query21	210	104	105	104
query22	4707	4460	4418	4418
query23	35230	34685	34860	34685
query24	5552	2611	2575	2575
query25	523	412	433	412
query26	643	160	155	155
query27	1732	295	291	291
query28	4689	2478	2481	2478
query29	699	442	427	427
query30	218	163	155	155
query31	1034	839	913	839
query32	64	58	58	58
query33	434	288	292	288
query34	933	514	529	514
query35	877	771	744	744
query36	1113	963	965	963
query37	123	77	70	70
query38	4554	4474	4556	4474
query39	1524	1474	1478	1474
query40	201	101	100	100
query41	44	44	39	39
query42	107	96	101	96
query43	540	504	506	504
query44	1215	800	810	800
query45	187	179	168	168
query46	1184	716	750	716
query47	2019	1932	1934	1932
query48	406	318	318	318
query49	727	397	377	377
query50	863	397	397	397
query51	7335	7283	7246	7246
query52	101	86	86	86
query53	255	185	177	177
query54	499	408	393	393
query55	79	75	77	75
query56	246	233	221	221
query57	1276	1122	1130	1122
query58	215	205	216	205
query59	3325	3061	3040	3040
query60	265	241	236	236
query61	111	106	104	104
query62	779	672	677	672
query63	226	197	196	196
query64	1420	646	660	646
query65	3319	3213	3241	3213
query66	691	299	300	299
query67	16046	15848	15660	15660
query68	3878	574	608	574
query69	448	259	251	251
query70	1180	1170	1170	1170
query71	341	248	249	248
query72	6357	4112	3986	3986
query73	760	365	371	365
query74	10271	9067	9144	9067
query75	3429	2707	2622	2622
query76	1902	1007	1099	1007
query77	500	283	292	283
query78	10377	9526	9511	9511
query79	1381	594	643	594
query80	879	419	439	419
query81	504	231	236	231
query82	1306	117	120	117
query83	248	143	146	143
query84	286	65	74	65
query85	949	302	297	297
query86	336	291	305	291
query87	4726	4624	4539	4539
query88	3547	2201	2172	2172
query89	421	294	291	291
query90	2012	190	192	190
query91	134	105	101	101
query92	69	49	47	47
query93	1871	555	541	541
query94	780	290	297	290
query95	354	251	249	249
query96	618	273	279	273
query97	2888	2767	2653	2653
query98	215	199	194	194
query99	1804	1328	1295	1295
Total cold run time: 319387 ms
Total hot run time: 198802 ms

doris-robot · 2024-12-02T10:41:29Z

ClickBench: Total hot run time: 32.8 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 817f076163b0dbf1003b3eb7240e530147b68635, data reload: false

query1	0.03	0.03	0.02
query2	0.06	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.11	0.10
query5	0.44	0.42	0.39
query6	1.16	0.65	0.66
query7	0.02	0.02	0.01
query8	0.04	0.03	0.02
query9	0.58	0.51	0.52
query10	0.54	0.57	0.56
query11	0.15	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.62	0.60
query14	2.71	2.72	2.73
query15	0.90	0.84	0.83
query16	0.40	0.39	0.37
query17	1.09	1.10	1.06
query18	0.22	0.20	0.22
query19	1.96	1.79	1.99
query20	0.02	0.01	0.01
query21	15.36	0.59	0.59
query22	2.45	2.53	1.89
query23	17.22	0.89	0.71
query24	2.76	1.55	1.19
query25	0.21	0.13	0.22
query26	0.41	0.14	0.14
query27	0.03	0.04	0.05
query28	10.21	1.11	1.08
query29	12.57	3.24	3.18
query30	0.24	0.08	0.07
query31	2.84	0.40	0.38
query32	3.28	0.46	0.47
query33	3.12	3.03	3.07
query34	16.99	4.48	4.51
query35	4.56	4.53	4.49
query36	0.64	0.48	0.48
query37	0.10	0.06	0.06
query38	0.05	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.13	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.31 s
Total hot run time: 32.8 s

github-actions · 2024-12-02T11:10:10Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-12-02T11:10:12Z

PR approved by anyone and no changes requested.

### What problem does this PR solve? Problem Summary: Fix mv rewrite fail when base table add column Such as mv def is as following: select o_orderstatus, l_suppkey, sum_merge(sum_state(o_shippriority)), group_concat_merge(group_concat_state(o_orderstatus)), avg_merge(avg_state(l_linenumber)), max_by_merge(max_by_state(O_COMMENT,o_totalprice)), count_merge(count_state(l_orderkey)), multi_distinct_count_merge(multi_distinct_count_state(l_shipmode)) from lineitem left join orders on l_orderkey = o_orderkey and l_shipdate = o_orderdate group by o_orderstatus, l_suppkey order by o_orderstatus; if add column to table orders, would cause rewrite fail ### Release note Fix mv rewrite fail when base table add column

…#44867 (#44956) Cherry-picked from #44867 Co-authored-by: seawinde <wusi@selectdb.com>

…#44867 (#44955) Cherry-picked from #44867 Co-authored-by: seawinde <wusi@selectdb.com>

[fix](mtmv) Fix mv rewrite fail when base table add column

817f076

morrySnow added dev/2.1.x dev/3.0.x labels Dec 2, 2024

morrySnow approved these changes Dec 2, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 2, 2024

github-actions bot added the reviewed label Dec 2, 2024

zddr approved these changes Dec 3, 2024

View reviewed changes

morrySnow merged commit 4b84de4 into apache:master Dec 3, 2024
27 of 28 checks passed

github-actions bot mentioned this pull request Dec 3, 2024

branch-3.0: [fix](mtmv) Fix mv rewrite fail when base table add column #44867 #44955

Merged

github-actions bot mentioned this pull request Dec 3, 2024

branch-2.1: [fix](mtmv) Fix mv rewrite fail when base table add column #44867 #44956

Merged

yiguolei pushed a commit that referenced this pull request Dec 4, 2024

branch-2.1: [fix](mtmv) Fix mv rewrite fail when base table add column …

5c686c3

…#44867 (#44956) Cherry-picked from #44867 Co-authored-by: seawinde <wusi@selectdb.com>

yiguolei added dev/2.1.8-merged and removed dev/2.1.x labels Dec 4, 2024

morrySnow pushed a commit that referenced this pull request Dec 6, 2024

branch-3.0: [fix](mtmv) Fix mv rewrite fail when base table add column …

0734cc3

…#44867 (#44955) Cherry-picked from #44867 Co-authored-by: seawinde <wusi@selectdb.com>

morrySnow added dev/3.0.4-merged and removed dev/3.0.x labels Dec 6, 2024

yiguolei mentioned this pull request Jan 19, 2025

Release Note 2.1.8 #47198

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](mtmv) Fix mv rewrite fail when base table add column #44867

[fix](mtmv) Fix mv rewrite fail when base table add column #44867

seawinde commented Dec 2, 2024

doris-robot commented Dec 2, 2024

seawinde commented Dec 2, 2024

doris-robot commented Dec 2, 2024

doris-robot commented Dec 2, 2024

doris-robot commented Dec 2, 2024

github-actions bot commented Dec 2, 2024

github-actions bot commented Dec 2, 2024

[fix](mtmv) Fix mv rewrite fail when base table add column #44867

[fix](mtmv) Fix mv rewrite fail when base table add column #44867

Conversation

seawinde commented Dec 2, 2024

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

doris-robot commented Dec 2, 2024

seawinde commented Dec 2, 2024

doris-robot commented Dec 2, 2024

doris-robot commented Dec 2, 2024

doris-robot commented Dec 2, 2024

github-actions bot commented Dec 2, 2024

github-actions bot commented Dec 2, 2024