Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067

Merged
merged 4 commits into from
Jul 3, 2024
Merged

[Fix](schema change) Fix can't do reorder column schema change for MOW table and duplicate key table #37067

merged 4 commits into from
Jul 3, 2024

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jul 1, 2024 •
edited
Loading

Proposed changes

  1. set column's field isAggregationTypeImplicit correctly for mow table and duplicate key table to let SchemaChangeHandler::createJob can recognize that the schema change is a column reorder case.
  2. remove useless test cases test_schema_change_datev2_with_delete, test_dup_keys_schema_change_datev2. These two cases wanted to test schema change from datev1/datetimev1 to datev2/datetimev2. But the default type of columns created as date/datetime are already datev2/datetimev2 now. Thest test cases should have met error "Nothing is changed. please check your alter stmt." when doris enable datev2/datetimev2 by default but it didn't because isAggregationTypeImplicit is wrongly set for duplicate table column.

branch-2.1-pick: #37226
branch-2.0-pick: #37515

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39870 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 427d676f3789f636ed5cb858ba01231e20c108e1, data reload: false

------ Round 1 ----------------------------------
q1	17605	4526	4253	4253
q2	2028	190	196	190
q3	10457	1176	1145	1145
q4	10187	811	817	811
q5	7479	2676	2629	2629
q6	225	137	142	137
q7	960	584	595	584
q8	9214	2082	2076	2076
q9	8884	6502	6523	6502
q10	8975	3707	3743	3707
q11	448	237	233	233
q12	445	226	230	226
q13	17771	3004	2972	2972
q14	266	224	218	218
q15	534	491	476	476
q16	526	384	375	375
q17	984	681	664	664
q18	8013	7410	7338	7338
q19	6495	1527	1437	1437
q20	655	329	335	329
q21	4965	3224	3917	3224
q22	411	344	345	344
Total cold run time: 117527 ms
Total hot run time: 39870 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4387	4218	4208	4208
q2	375	259	258	258
q3	2959	2874	2881	2874
q4	2018	1706	1705	1705
q5	5627	5524	5497	5497
q6	229	130	139	130
q7	2204	1874	1823	1823
q8	3323	3418	3425	3418
q9	8695	8663	8857	8663
q10	4178	3855	3701	3701
q11	606	500	498	498
q12	832	627	625	625
q13	15916	3130	3136	3130
q14	303	268	308	268
q15	540	478	498	478
q16	477	421	422	421
q17	1817	1523	1512	1512
q18	8076	7981	7889	7889
q19	1885	1497	1648	1497
q20	2184	1850	1858	1850
q21	5054	4944	4684	4684
q22	670	569	564	564
Total cold run time: 72355 ms
Total hot run time: 55693 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172284 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 427d676f3789f636ed5cb858ba01231e20c108e1, data reload: false

query1	928	382	370	370
query2	6448	2408	2345	2345
query3	6643	211	215	211
query4	18937	17316	17178	17178
query5	3722	482	480	480
query6	262	164	182	164
query7	4572	286	285	285
query8	299	272	272	272
query9	8612	2406	2388	2388
query10	570	314	287	287
query11	10569	10000	9924	9924
query12	121	87	84	84
query13	1649	371	363	363
query14	10090	7730	6126	6126
query15	221	186	181	181
query16	7814	288	272	272
query17	1872	566	543	543
query18	1969	283	278	278
query19	198	155	157	155
query20	90	79	86	79
query21	213	131	125	125
query22	4176	4019	4128	4019
query23	33823	33797	33627	33627
query24	10674	2889	2869	2869
query25	604	379	373	373
query26	724	150	152	150
query27	2305	317	321	317
query28	6028	2184	2147	2147
query29	880	636	621	621
query30	261	160	171	160
query31	1030	761	768	761
query32	102	53	52	52
query33	665	293	287	287
query34	893	486	481	481
query35	750	668	660	660
query36	1121	978	977	977
query37	137	80	80	80
query38	2960	2847	2823	2823
query39	877	820	854	820
query40	209	140	126	126
query41	55	57	53	53
query42	110	98	105	98
query43	580	560	552	552
query44	1076	727	734	727
query45	199	164	159	159
query46	1071	716	702	702
query47	1811	1766	1777	1766
query48	357	289	297	289
query49	824	406	431	406
query50	768	377	382	377
query51	6902	6720	6847	6720
query52	105	95	89	89
query53	355	291	282	282
query54	874	440	432	432
query55	72	72	72	72
query56	278	257	253	253
query57	1139	1062	1034	1034
query58	238	242	252	242
query59	3338	3103	3290	3103
query60	306	284	269	269
query61	92	90	93	90
query62	599	439	438	438
query63	325	290	291	290
query64	8565	2247	1730	1730
query65	3156	3062	3111	3062
query66	752	332	319	319
query67	15301	14889	15059	14889
query68	6182	537	532	532
query69	675	420	326	326
query70	1179	1123	1167	1123
query71	440	270	273	270
query72	7577	5510	5398	5398
query73	791	332	320	320
query74	5938	5518	5428	5428
query75	3917	2648	2646	2646
query76	3774	951	906	906
query77	666	295	296	295
query78	10355	9852	9603	9603
query79	2778	510	517	510
query80	1397	466	459	459
query81	555	216	216	216
query82	893	105	104	104
query83	203	164	166	164
query84	276	90	88	88
query85	1248	277	272	272
query86	460	314	304	304
query87	3327	3072	3125	3072
query88	3503	2381	2375	2375
query89	475	378	384	378
query90	1738	182	181	181
query91	122	99	98	98
query92	59	51	48	48
query93	3618	499	492	492
query94	1125	185	180	180
query95	411	309	308	308
query96	608	269	263	263
query97	3221	3030	3004	3004
query98	215	197	196	196
query99	1251	842	822	822
Total cold run time: 272243 ms
Total hot run time: 172284 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 427d676f3789f636ed5cb858ba01231e20c108e1, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.05	0.04
query3	0.22	0.05	0.05
query4	1.67	0.06	0.07
query5	0.51	0.48	0.47
query6	1.13	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.50
query10	0.53	0.54	0.54
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.76	0.76
query15	0.83	0.81	0.82
query16	0.35	0.37	0.37
query17	0.95	1.02	1.03
query18	0.22	0.26	0.25
query19	1.88	1.80	1.69
query20	0.02	0.01	0.01
query21	15.45	0.76	0.65
query22	4.22	7.66	1.77
query23	18.27	1.37	1.30
query24	2.14	0.23	0.23
query25	0.17	0.09	0.08
query26	0.26	0.18	0.18
query27	0.09	0.07	0.08
query28	13.24	1.03	0.99
query29	12.62	3.30	3.28
query30	0.25	0.06	0.06
query31	2.85	0.41	0.40
query32	3.24	0.46	0.49
query33	2.86	2.85	2.87
query34	17.09	4.41	4.43
query35	4.48	4.48	4.52
query36	0.65	0.46	0.48
query37	0.19	0.15	0.16
query38	0.15	0.15	0.15
query39	0.05	0.03	0.04
query40	0.17	0.16	0.16
query41	0.09	0.04	0.06
query42	0.06	0.05	0.05
query43	0.05	0.04	0.03
Total cold run time: 109.37 s
Total hot run time: 30.46 s

@Lchangliang
Copy link
Contributor

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by anyone and no changes requested.

dataroaring
dataroaring previously approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
zhannngchen
zhannngchen previously approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen
Copy link
Contributor

run p0

cjj2010
cjj2010 approved these changes Jul 2, 2024
View reviewed changes
Copy link
Contributor

@cjj2010 cjj2010 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bobhan1 bobhan1 dismissed stale reviews from zhannngchen and dataroaring via 8505246 July 2, 2024 07:16
@bobhan1 bobhan1 force-pushed the fix-reorder-struct branch from 427d676 to 8505246 Compare July 2, 2024 07:16
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 2, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 2, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40197 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 850524662e168813180958cba8017512888bbb5c, data reload: false

------ Round 1 ----------------------------------
q1	17600	4459	4330	4330
q2	2006	198	211	198
q3	10444	1244	1192	1192
q4	10194	812	828	812
q5	7476	2693	2772	2693
q6	221	137	137	137
q7	963	595	601	595
q8	9230	2097	2070	2070
q9	8867	6504	6500	6500
q10	8963	3753	3755	3753
q11	442	238	237	237
q12	443	240	241	240
q13	17853	2964	2973	2964
q14	287	240	228	228
q15	526	487	494	487
q16	515	388	375	375
q17	984	616	752	616
q18	8177	7448	7391	7391
q19	7659	1493	1512	1493
q20	664	327	324	324
q21	5003	3216	3975	3216
q22	411	346	353	346
Total cold run time: 118928 ms
Total hot run time: 40197 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4273	4271	4271
q2	385	274	276	274
q3	3120	2904	2957	2904
q4	1943	1763	1714	1714
q5	5565	5484	5466	5466
q6	230	134	141	134
q7	2297	1843	1873	1843
q8	3304	3479	3486	3479
q9	8786	8817	8712	8712
q10	4177	3753	3846	3753
q11	588	495	501	495
q12	815	660	640	640
q13	17231	3161	3181	3161
q14	310	268	281	268
q15	522	484	498	484
q16	486	434	426	426
q17	1830	1548	1502	1502
q18	8059	7964	7851	7851
q19	1877	1600	1574	1574
q20	2958	1862	1857	1857
q21	5148	4707	4792	4707
q22	630	573	580	573
Total cold run time: 74701 ms
Total hot run time: 56088 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173890 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 850524662e168813180958cba8017512888bbb5c, data reload: false

query1	906	387	376	376
query2	6466	2491	2352	2352
query3	6638	223	221	221
query4	20688	17592	17356	17356
query5	3618	475	502	475
query6	256	167	176	167
query7	4608	289	285	285
query8	321	296	296	296
query9	8526	2353	2340	2340
query10	562	292	276	276
query11	10382	9890	9935	9890
query12	113	89	80	80
query13	1647	359	363	359
query14	10274	7693	7648	7648
query15	232	184	188	184
query16	7641	270	256	256
query17	1892	577	513	513
query18	1828	268	273	268
query19	195	151	153	151
query20	86	84	79	79
query21	212	134	122	122
query22	4290	3960	4169	3960
query23	34203	33766	33654	33654
query24	11050	2891	2947	2891
query25	663	393	387	387
query26	1150	166	160	160
query27	2479	330	324	324
query28	6902	2122	2109	2109
query29	943	610	640	610
query30	234	157	161	157
query31	971	760	742	742
query32	97	56	55	55
query33	751	296	286	286
query34	995	481	485	481
query35	762	624	668	624
query36	1143	978	974	974
query37	163	76	81	76
query38	2928	2780	2775	2775
query39	897	786	819	786
query40	212	123	125	123
query41	54	53	58	53
query42	128	104	102	102
query43	600	565	570	565
query44	1228	746	724	724
query45	194	169	165	165
query46	1089	731	720	720
query47	1837	1776	1749	1749
query48	367	305	304	304
query49	865	413	425	413
query50	775	379	385	379
query51	6887	6779	6693	6693
query52	103	100	98	98
query53	374	314	309	309
query54	892	449	441	441
query55	80	76	76	76
query56	299	286	279	279
query57	1121	1047	1039	1039
query58	261	248	253	248
query59	3431	3141	3386	3141
query60	316	283	277	277
query61	91	91	101	91
query62	606	434	440	434
query63	321	284	295	284
query64	8842	2253	1749	1749
query65	3212	3057	3102	3057
query66	710	321	325	321
query67	15571	15291	14799	14799
query68	4556	531	532	531
query69	601	358	319	319
query70	1163	1164	1116	1116
query71	401	284	279	279
query72	8063	5595	5387	5387
query73	743	321	321	321
query74	5871	5507	5395	5395
query75	3412	2676	2682	2676
query76	2978	961	953	953
query77	599	295	294	294
query78	10474	9828	9696	9696
query79	2307	519	507	507
query80	1118	463	476	463
query81	556	215	224	215
query82	776	108	103	103
query83	317	172	173	172
query84	252	89	86	86
query85	2005	285	288	285
query86	479	314	312	312
query87	3308	3088	3040	3040
query88	4305	2378	2353	2353
query89	474	394	376	376
query90	1950	192	191	191
query91	130	99	102	99
query92	61	52	47	47
query93	2313	498	491	491
query94	1243	240	184	184
query95	410	314	317	314
query96	597	264	266	264
query97	3267	3013	3010	3010
query98	226	205	200	200
query99	1151	833	835	833
Total cold run time: 274284 ms
Total hot run time: 173890 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.4 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 850524662e168813180958cba8017512888bbb5c, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.22	0.04	0.05
query4	1.68	0.10	0.08
query5	0.49	0.49	0.49
query6	1.13	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.48	0.48
query10	0.55	0.55	0.56
query11	0.15	0.12	0.11
query12	0.16	0.12	0.12
query13	0.60	0.58	0.59
query14	0.78	0.77	0.78
query15	0.84	0.82	0.81
query16	0.37	0.38	0.37
query17	0.96	1.03	0.99
query18	0.23	0.25	0.22
query19	1.79	1.66	1.71
query20	0.01	0.02	0.02
query21	15.43	0.76	0.64
query22	4.38	7.59	1.87
query23	18.31	1.33	1.25
query24	2.05	0.22	0.22
query25	0.15	0.09	0.08
query26	0.28	0.17	0.18
query27	0.08	0.08	0.08
query28	13.32	1.01	1.01
query29	12.60	3.25	3.25
query30	0.26	0.06	0.05
query31	2.87	0.39	0.38
query32	3.27	0.48	0.47
query33	2.85	2.89	2.96
query34	16.99	4.42	4.41
query35	4.46	4.44	4.57
query36	0.64	0.45	0.49
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.04	0.03	0.03
query40	0.17	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.38 s
Total hot run time: 30.4 s

@bobhan1 bobhan1 force-pushed the fix-reorder-struct branch from ee047b0 to f0a7810 Compare July 2, 2024 13:05
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 2, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39556 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f0a7810ef52c84534f0d0dc3cac84fdedd95d42c, data reload: false

------ Round 1 ----------------------------------
q1	17610	4291	4237	4237
q2	2015	191	182	182
q3	10474	1133	996	996
q4	10196	775	758	758
q5	7487	2668	2641	2641
q6	220	139	140	139
q7	955	595	598	595
q8	9215	2065	2054	2054
q9	8918	6508	6445	6445
q10	8970	3705	3669	3669
q11	452	236	233	233
q12	480	245	228	228
q13	17835	2951	2958	2951
q14	261	225	232	225
q15	514	490	491	490
q16	524	385	371	371
q17	957	661	656	656
q18	8004	7514	7305	7305
q19	5498	1483	1451	1451
q20	687	322	324	322
q21	4884	3268	3969	3268
q22	407	340	343	340
Total cold run time: 116563 ms
Total hot run time: 39556 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4254	4184	4184
q2	367	275	257	257
q3	2973	2762	2827	2762
q4	2014	1764	1697	1697
q5	5698	5494	5461	5461
q6	228	131	130	130
q7	2165	1828	1901	1828
q8	3248	3405	3378	3378
q9	8647	8595	8742	8595
q10	4040	3889	3795	3795
q11	576	483	479	479
q12	785	637	638	637
q13	17158	3137	3175	3137
q14	300	255	297	255
q15	528	510	488	488
q16	501	417	425	417
q17	1833	1523	1500	1500
q18	7925	7802	7916	7802
q19	1879	1476	1581	1476
q20	2170	1917	1870	1870
q21	10909	5047	4850	4850
q22	640	532	541	532
Total cold run time: 78950 ms
Total hot run time: 55530 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172920 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f0a7810ef52c84534f0d0dc3cac84fdedd95d42c, data reload: false

query1	908	390	383	383
query2	7355	2479	2427	2427
query3	6635	214	222	214
query4	19258	17382	17128	17128
query5	3714	478	470	470
query6	266	176	185	176
query7	4598	294	294	294
query8	326	296	308	296
query9	8762	2361	2361	2361
query10	553	314	278	278
query11	10478	9998	9981	9981
query12	119	83	88	83
query13	1641	362	369	362
query14	9433	6888	7678	6888
query15	236	182	182	182
query16	7785	261	273	261
query17	1894	551	515	515
query18	1964	271	267	267
query19	191	153	148	148
query20	91	80	80	80
query21	202	136	125	125
query22	4312	3963	3977	3963
query23	33870	33604	33521	33521
query24	11117	2920	2905	2905
query25	641	376	387	376
query26	1117	156	156	156
query27	2277	335	328	328
query28	6938	2087	2084	2084
query29	909	636	633	633
query30	278	172	160	160
query31	951	753	746	746
query32	91	53	53	53
query33	748	288	294	288
query34	984	478	478	478
query35	728	629	640	629
query36	1139	984	996	984
query37	146	88	77	77
query38	2874	2832	2822	2822
query39	877	834	824	824
query40	211	130	126	126
query41	52	57	55	55
query42	108	101	106	101
query43	587	577	564	564
query44	1175	714	715	714
query45	192	158	167	158
query46	1066	717	724	717
query47	1828	1752	1772	1752
query48	381	285	300	285
query49	849	403	414	403
query50	769	382	380	380
query51	6885	6770	6687	6687
query52	99	97	100	97
query53	362	292	292	292
query54	868	432	442	432
query55	74	74	73	73
query56	289	261	265	261
query57	1111	1028	1067	1028
query58	262	253	261	253
query59	3491	3158	3354	3158
query60	321	289	297	289
query61	115	115	115	115
query62	601	450	431	431
query63	322	290	302	290
query64	8904	2245	1747	1747
query65	3117	3100	3122	3100
query66	746	318	326	318
query67	15524	15073	14872	14872
query68	5628	525	533	525
query69	628	442	324	324
query70	1130	1119	1153	1119
query71	462	289	275	275
query72	7597	5118	5053	5053
query73	758	328	327	327
query74	6181	5518	5488	5488
query75	3817	2652	2674	2652
query76	3641	922	907	907
query77	638	295	293	293
query78	10466	9835	9722	9722
query79	3329	510	573	510
query80	2172	474	476	474
query81	590	216	220	216
query82	1474	112	107	107
query83	321	169	168	168
query84	266	91	84	84
query85	1409	290	290	290
query86	472	313	312	312
query87	3313	3078	3095	3078
query88	4106	2368	2354	2354
query89	482	402	382	382
query90	1770	188	178	178
query91	135	103	99	99
query92	59	49	48	48
query93	4474	492	498	492
query94	1066	184	186	184
query95	403	311	312	311
query96	607	266	267	266
query97	3171	3001	3047	3001
query98	207	202	200	200
query99	1157	837	842	837
Total cold run time: 278644 ms
Total hot run time: 172920 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f0a7810ef52c84534f0d0dc3cac84fdedd95d42c, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.08
query5	0.50	0.49	0.49
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.48
query10	0.57	0.58	0.55
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.59
query14	0.78	0.77	0.78
query15	0.85	0.81	0.81
query16	0.37	0.36	0.35
query17	1.05	0.98	0.97
query18	0.25	0.25	0.25
query19	1.89	1.69	1.70
query20	0.02	0.01	0.01
query21	15.43	0.81	0.67
query22	4.07	7.84	1.50
query23	18.26	1.38	1.29
query24	2.11	0.23	0.22
query25	0.15	0.09	0.09
query26	0.26	0.18	0.17
query27	0.08	0.08	0.08
query28	13.32	1.02	1.01
query29	12.57	3.31	3.28
query30	0.25	0.06	0.05
query31	2.86	0.38	0.39
query32	3.30	0.48	0.46
query33	2.89	2.98	2.98
query34	17.08	4.40	4.43
query35	4.44	4.46	4.47
query36	0.65	0.46	0.50
query37	0.19	0.16	0.15
query38	0.16	0.15	0.14
query39	0.04	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.4 s
Total hot run time: 30.24 s

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 3, 2024

run external

@bobhan1 bobhan1 force-pushed the fix-reorder-struct branch from f0a7810 to fb24d58 Compare July 3, 2024 01:47
@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40679 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fb24d58e23b7b9f8e5168d0083d6e19a6fc28240, data reload: false

------ Round 1 ----------------------------------
q1	17986	4966	4392	4392
q2	2240	190	190	190
q3	10534	1189	1125	1125
q4	10189	850	785	785
q5	7585	2623	2635	2623
q6	222	142	139	139
q7	957	592	611	592
q8	9216	2078	2043	2043
q9	8733	6525	6476	6476
q10	8863	3747	3742	3742
q11	456	246	243	243
q12	408	235	234	234
q13	17772	2975	3031	2975
q14	272	238	226	226
q15	520	482	478	478
q16	503	383	379	379
q17	978	591	645	591
q18	8062	7480	7423	7423
q19	1851	1570	1421	1421
q20	677	331	316	316
q21	5023	3980	3953	3953
q22	404	341	333	333
Total cold run time: 113451 ms
Total hot run time: 40679 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4311	4297	4242	4242
q2	371	270	264	264
q3	3016	2712	2748	2712
q4	1910	1649	1634	1634
q5	5223	5264	5260	5260
q6	219	136	134	134
q7	2153	1794	1707	1707
q8	3182	3321	3288	3288
q9	8327	8283	8347	8283
q10	3931	3695	3650	3650
q11	571	479	491	479
q12	770	607	616	607
q13	16478	3031	2993	2993
q14	279	269	247	247
q15	508	472	477	472
q16	486	415	430	415
q17	1780	1514	1454	1454
q18	7642	7548	7449	7449
q19	2609	1591	1578	1578
q20	1997	1785	1781	1781
q21	4796	4707	4662	4662
q22	619	562	544	544
Total cold run time: 71178 ms
Total hot run time: 53855 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173581 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fb24d58e23b7b9f8e5168d0083d6e19a6fc28240, data reload: false

query1	937	371	373	371
query2	6455	2322	2219	2219
query3	6662	211	219	211
query4	19390	17403	17439	17403
query5	4180	486	472	472
query6	268	164	175	164
query7	4591	294	296	294
query8	309	273	273	273
query9	8510	2381	2369	2369
query10	624	298	278	278
query11	10557	10168	10046	10046
query12	129	92	86	86
query13	1666	377	379	377
query14	9620	7838	7738	7738
query15	229	186	184	184
query16	7822	275	269	269
query17	1892	546	547	546
query18	1924	281	276	276
query19	200	150	156	150
query20	95	83	86	83
query21	214	147	142	142
query22	4358	4103	4127	4103
query23	33863	33116	32880	32880
query24	11216	2942	2765	2765
query25	633	354	361	354
query26	1600	153	151	151
query27	2966	316	319	316
query28	7218	2066	2055	2055
query29	1015	648	605	605
query30	278	156	157	156
query31	956	731	764	731
query32	93	54	56	54
query33	781	287	277	277
query34	961	471	470	470
query35	737	657	626	626
query36	1105	928	915	915
query37	150	74	78	74
query38	2891	2763	2759	2759
query39	843	787	783	783
query40	281	121	125	121
query41	52	52	54	52
query42	119	100	103	100
query43	571	526	532	526
query44	1236	735	744	735
query45	194	164	159	159
query46	1081	725	749	725
query47	1842	1756	1778	1756
query48	381	306	301	301
query49	1162	423	418	418
query50	782	381	389	381
query51	6913	6774	6704	6704
query52	106	93	97	93
query53	367	300	298	298
query54	992	454	444	444
query55	74	74	76	74
query56	303	258	269	258
query57	1132	1045	1053	1045
query58	244	230	267	230
query59	3435	2955	3025	2955
query60	300	275	274	274
query61	94	93	116	93
query62	644	447	434	434
query63	327	300	295	295
query64	9832	2207	1773	1773
query65	3165	3099	3090	3090
query66	1383	343	322	322
query67	15409	15090	14925	14925
query68	6269	555	542	542
query69	668	415	313	313
query70	1111	1179	1145	1145
query71	488	279	277	277
query72	7291	5527	5702	5527
query73	801	321	328	321
query74	5867	5563	5496	5496
query75	3998	2703	2672	2672
query76	4378	931	971	931
query77	645	310	306	306
query78	10523	10022	9810	9810
query79	2766	505	509	505
query80	1530	468	467	467
query81	589	215	222	215
query82	892	108	106	106
query83	235	174	167	167
query84	278	85	88	85
query85	1451	285	267	267
query86	468	332	335	332
query87	3327	3045	3071	3045
query88	3880	2437	2440	2437
query89	461	366	370	366
query90	1864	183	182	182
query91	121	96	96	96
query92	65	52	49	49
query93	3198	524	524	524
query94	1162	194	187	187
query95	403	311	316	311
query96	611	275	269	269
query97	3168	2990	3033	2990
query98	233	205	200	200
query99	1233	835	849	835
Total cold run time: 280333 ms
Total hot run time: 173581 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fb24d58e23b7b9f8e5168d0083d6e19a6fc28240, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.06
query5	0.51	0.48	0.48
query6	1.15	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.47
query10	0.55	0.54	0.54
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.80	0.78	0.79
query15	0.83	0.82	0.83
query16	0.36	0.38	0.37
query17	1.04	0.98	0.95
query18	0.21	0.28	0.24
query19	1.87	1.72	1.77
query20	0.02	0.02	0.01
query21	15.45	0.76	0.66
query22	4.35	7.04	2.24
query23	18.30	1.41	1.25
query24	1.94	0.25	0.23
query25	0.17	0.09	0.08
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.28	1.04	1.00
query29	12.61	3.23	3.22
query30	0.26	0.06	0.06
query31	2.92	0.38	0.39
query32	3.24	0.46	0.46
query33	2.90	2.92	2.95
query34	17.04	4.45	4.49
query35	4.50	4.53	4.48
query36	0.66	0.47	0.49
query37	0.19	0.16	0.16
query38	0.15	0.15	0.16
query39	0.04	0.04	0.04
query40	0.18	0.15	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.06
query43	0.04	0.03	0.04
Total cold run time: 109.58 s
Total hot run time: 30.93 s

zhannngchen
zhannngchen approved these changes Jul 3, 2024
View reviewed changes
Copy link
Contributor

@zhannngchen zhannngchen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhannngchen zhannngchen merged commit 4984222 into apache:master Jul 3, 2024
25 of 27 checks passed
@bobhan1 bobhan1 mentioned this pull request Jul 3, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 3, 2024
@bobhan1
[branch-2.1] Picks "[Fix](schema change) Fix can't do reorder column …
69aebc2 
…schema change for MOW table and duplicate key table #37067" (#37226)

## Proposed changes

picks #37067
@bobhan1 bobhan1 mentioned this pull request Jul 4, 2024
Merged
zhannngchen pushed a commit that referenced this pull request Jul 4, 2024
@bobhan1
[fix](regression) Fix p0 case test_modify_reorder_column (#37256)
9ce5707 
#37067 passed all pipeline cases
based on the branch that doesn't contain
#37039 which changed the output
format merged later, causing this case's failure.
@bobhan1 bobhan1 mentioned this pull request Jul 9, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 13, 2024
@bobhan1
[branch-2.0] Picks "[Fix](schema change) Fix can't do reorder column …
4b85545 
…schema change for MOW table and duplicate key table #37067" (#37515)

## Proposed changes

picks #37067
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@bobhan1 @dataroaring
[Fix](schema change) Fix can't do reorder column schema change for MO…
b7d10a3 
…W table and duplicate key table (#37067)

## Proposed changes

1. set column's field `isAggregationTypeImplicit` correctly for mow
table and duplicate key table to let `SchemaChangeHandler::createJob`
can recognize that the schema change is a column reorder case.
2. remove useless test cases `test_schema_change_datev2_with_delete`,
`test_dup_keys_schema_change_datev2`. These two cases wanted to test
schema change from `datev1`/`datetimev1` to `datev2`/`datetimev2`. But
the default type of columns created as `date`/`datetime` are already
`datev2`/`datetimev2` now. Thest test cases should have met error
"Nothing is changed. please check your alter stmt." when doris enable
`datev2`/`datetimev2` by default but it didn't because
`isAggregationTypeImplicit` is wrongly set for duplicate table column.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@bobhan1 @dataroaring
[fix](regression) Fix p0 case test_modify_reorder_column (#37256)
69a0ff6 
#37067 passed all pipeline cases
based on the branch that doesn't contain
#37039 which changed the output
format merged later, causing this case's failure.
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@bobhan1
[branch-2.0] Picks "[Fix](schema change) Fix can't do reorder column …
36c7fa2 
…schema change for MOW table and duplicate key table apache#37067" (apache#37515)

## Proposed changes

picks apache#37067
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@zhannngchen zhannngchen zhannngchen approved these changes

@cjj2010 cjj2010 cjj2010 approved these changes

@Lchangliang Lchangliang Lchangliang approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@bobhan1 @doris-robot @Lchangliang @zhannngchen @cjj2010 @dataroaring