Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](multi-catalog)fix paimon meta properties convert #37249

Merged
merged 9 commits into from
Jul 16, 2024
Merged

[fix](multi-catalog)fix paimon meta properties convert #37249

merged 9 commits into from
Jul 16, 2024

Conversation

wsjz
Copy link
Contributor

@wsjz wsjz commented Jul 3, 2024 •
edited
Loading

Proposed changes

  1. set default region to 'us-east-1' if no any region can be read
  2. add meta properties convert to paimon
  3. recognize ip endpoint, and return null for getRegionOfEndpoint method
  4. add ut case for meta properties convert

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@wsjz wsjz force-pushed the fix_paimon_props branch from d273d7a to b2dc07d Compare July 3, 2024 15:55
@wsjz wsjz marked this pull request as ready for review July 3, 2024 15:57
@wsjz wsjz force-pushed the fix_paimon_props branch 2 times, most recently from 0b5c29b to c8c3162 Compare July 4, 2024 07:45
@wsjz
Copy link
Contributor Author

wsjz commented Jul 4, 2024

run buildall

1 similar comment
@wsjz
Copy link
Contributor Author

wsjz commented Jul 4, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39898 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c8c316238cac39cd6acb32310490c31b944416cf, data reload: false

------ Round 1 ----------------------------------
q1	17729	4332	4285	4285
q2	2048	197	192	192
q3	10504	1224	1038	1038
q4	10195	774	781	774
q5	7520	2677	2622	2622
q6	223	140	139	139
q7	955	605	604	604
q8	9239	2098	2077	2077
q9	8984	6477	6486	6477
q10	9009	3699	3692	3692
q11	450	241	231	231
q12	502	237	233	233
q13	18976	2948	2994	2948
q14	271	217	226	217
q15	520	492	484	484
q16	504	396	380	380
q17	975	633	710	633
q18	7998	7452	7493	7452
q19	3905	1544	1524	1524
q20	685	321	339	321
q21	5002	3234	3343	3234
q22	394	349	341	341
Total cold run time: 116588 ms
Total hot run time: 39898 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4369	4241	4241	4241
q2	373	263	266	263
q3	2967	2767	2866	2767
q4	2015	1800	1705	1705
q5	5660	5549	5506	5506
q6	215	134	129	129
q7	2179	1862	1871	1862
q8	3298	3431	3397	3397
q9	8678	8695	8795	8695
q10	4144	3948	3820	3820
q11	594	501	498	498
q12	818	637	672	637
q13	16006	3169	3205	3169
q14	298	287	277	277
q15	540	500	490	490
q16	474	445	429	429
q17	1788	1531	1516	1516
q18	8110	8121	7832	7832
q19	6485	1607	1580	1580
q20	2052	1918	1832	1832
q21	5129	4986	4862	4862
q22	637	561	556	556
Total cold run time: 76829 ms
Total hot run time: 56063 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171773 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c8c316238cac39cd6acb32310490c31b944416cf, data reload: false

query1	920	384	364	364
query2	6457	2526	2263	2263
query3	6638	212	213	212
query4	20484	17537	17383	17383
query5	3645	495	494	494
query6	261	170	180	170
query7	4600	306	287	287
query8	315	302	288	288
query9	8547	2411	2420	2411
query10	568	298	277	277
query11	10636	10037	10060	10037
query12	114	96	85	85
query13	1635	392	357	357
query14	8707	5924	6201	5924
query15	226	191	183	183
query16	6976	305	303	303
query17	926	556	514	514
query18	1847	269	277	269
query19	190	147	154	147
query20	89	89	86	86
query21	214	132	125	125
query22	4365	4136	4034	4034
query23	33935	33639	33623	33623
query24	11865	2921	2772	2772
query25	599	381	387	381
query26	728	152	151	151
query27	2596	321	326	321
query28	6984	2152	2141	2141
query29	939	689	644	644
query30	263	154	162	154
query31	980	750	749	749
query32	95	59	62	59
query33	761	292	286	286
query34	1110	472	480	472
query35	742	688	665	665
query36	1166	967	993	967
query37	133	81	75	75
query38	2947	2875	2815	2815
query39	901	842	852	842
query40	228	133	129	129
query41	70	54	51	51
query42	122	98	103	98
query43	612	556	554	554
query44	1195	744	723	723
query45	195	168	166	166
query46	1088	740	701	701
query47	1877	1778	1771	1771
query48	368	316	309	309
query49	874	426	447	426
query50	770	395	387	387
query51	6869	6891	6788	6788
query52	108	87	94	87
query53	356	293	286	286
query54	893	439	434	434
query55	75	71	72	71
query56	283	263	271	263
query57	1132	1050	1083	1050
query58	254	248	248	248
query59	3525	3142	3302	3142
query60	310	277	275	275
query61	101	93	96	93
query62	606	434	444	434
query63	322	290	286	286
query64	8952	2304	1757	1757
query65	3154	3128	3132	3128
query66	930	328	326	326
query67	15357	14957	15116	14957
query68	4541	525	538	525
query69	453	318	314	314
query70	1220	1162	1128	1128
query71	403	276	277	276
query72	7029	5057	5764	5057
query73	749	318	323	318
query74	6021	5502	5535	5502
query75	3345	2685	2692	2685
query76	1997	904	874	874
query77	447	301	289	289
query78	10178	8887	8951	8887
query79	3364	517	521	517
query80	2068	461	462	461
query81	595	219	224	219
query82	738	109	103	103
query83	275	169	168	168
query84	263	90	86	86
query85	1892	274	344	274
query86	494	319	278	278
query87	3260	3122	3123	3122
query88	4150	2366	2363	2363
query89	503	386	382	382
query90	1787	182	188	182
query91	127	102	99	99
query92	64	49	50	49
query93	5021	496	498	496
query94	1157	216	210	210
query95	403	318	320	318
query96	597	269	260	260
query97	3234	3055	3036	3036
query98	220	209	196	196
query99	1206	834	829	829
Total cold run time: 273531 ms
Total hot run time: 171773 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.28 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c8c316238cac39cd6acb32310490c31b944416cf, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.48	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.56	0.48	0.50
query10	0.55	0.55	0.54
query11	0.15	0.11	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	0.76	0.79	0.77
query15	0.83	0.81	0.83
query16	0.37	0.37	0.35
query17	0.97	0.95	0.98
query18	0.23	0.26	0.23
query19	1.83	1.66	1.75
query20	0.01	0.00	0.01
query21	15.39	0.77	0.66
query22	4.16	8.02	1.75
query23	18.30	1.34	1.22
query24	2.08	0.23	0.23
query25	0.16	0.08	0.10
query26	0.27	0.17	0.17
query27	0.08	0.08	0.08
query28	13.27	1.01	1.01
query29	12.59	3.34	3.29
query30	0.26	0.06	0.05
query31	2.86	0.41	0.39
query32	3.27	0.47	0.46
query33	2.89	2.92	2.92
query34	17.09	4.39	4.44
query35	4.54	4.45	4.50
query36	0.65	0.48	0.50
query37	0.18	0.15	0.16
query38	0.16	0.15	0.15
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.32 s
Total hot run time: 30.28 s

@wsjz
Copy link
Contributor Author

wsjz commented Jul 5, 2024

run buildall

@morningman
Copy link
Contributor

run buildall

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40143 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7efd9c519cb039fa6957b752775589cdf288df11, data reload: false

------ Round 1 ----------------------------------
q1	17625	4358	4266	4266
q2	2016	192	186	186
q3	10452	1120	1061	1061
q4	10193	826	774	774
q5	7565	2661	2680	2661
q6	218	137	135	135
q7	953	606	589	589
q8	9211	2068	2054	2054
q9	8921	6599	6572	6572
q10	9103	3807	3844	3807
q11	445	245	245	245
q12	394	231	231	231
q13	18166	3011	3074	3011
q14	284	234	242	234
q15	544	525	487	487
q16	498	379	389	379
q17	964	665	688	665
q18	8140	7541	7441	7441
q19	7818	1524	1502	1502
q20	697	339	339	339
q21	4994	3224	3164	3164
q22	399	356	340	340
Total cold run time: 119600 ms
Total hot run time: 40143 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4450	4358	4311	4311
q2	375	291	289	289
q3	3338	3067	3073	3067
q4	2094	1785	1749	1749
q5	5749	5671	5789	5671
q6	257	145	147	145
q7	2340	1963	1936	1936
q8	3329	3437	3427	3427
q9	8884	8936	8732	8732
q10	4192	3755	3891	3755
q11	616	498	503	498
q12	825	627	660	627
q13	16342	3151	3201	3151
q14	322	279	283	279
q15	541	507	501	501
q16	510	450	441	441
q17	1835	1572	1506	1506
q18	8136	8184	7834	7834
q19	1805	1478	1523	1478
q20	2145	1899	1858	1858
q21	7857	5004	4805	4805
q22	626	565	541	541
Total cold run time: 76568 ms
Total hot run time: 56601 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 175147 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7efd9c519cb039fa6957b752775589cdf288df11, data reload: false

query1	906	386	365	365
query2	6464	2318	2353	2318
query3	6637	212	231	212
query4	28099	17459	17277	17277
query5	3612	467	473	467
query6	260	170	154	154
query7	4585	291	288	288
query8	323	303	309	303
query9	8501	2475	2448	2448
query10	438	285	272	272
query11	12517	10018	10044	10018
query12	113	80	84	80
query13	1641	367	355	355
query14	10205	7868	7787	7787
query15	244	198	185	185
query16	7557	326	325	325
query17	1777	562	525	525
query18	1569	314	295	295
query19	199	150	144	144
query20	88	81	83	81
query21	201	129	132	129
query22	4686	4178	4050	4050
query23	34296	33598	33684	33598
query24	10820	2924	2896	2896
query25	616	427	433	427
query26	722	165	157	157
query27	2281	277	279	277
query28	5857	2158	2155	2155
query29	892	674	653	653
query30	266	167	163	163
query31	1041	776	767	767
query32	101	55	60	55
query33	720	318	322	318
query34	898	491	506	491
query35	695	603	649	603
query36	1183	974	940	940
query37	139	82	84	82
query38	2955	2838	2825	2825
query39	889	819	821	819
query40	205	118	119	118
query41	53	52	55	52
query42	113	100	96	96
query43	588	560	566	560
query44	1140	737	719	719
query45	196	169	166	166
query46	1100	751	731	731
query47	1850	1762	1773	1762
query48	365	291	293	291
query49	823	410	412	410
query50	795	392	394	392
query51	6842	6853	6771	6771
query52	109	88	114	88
query53	368	289	293	289
query54	876	457	453	453
query55	79	78	74	74
query56	289	266	290	266
query57	1109	1079	1059	1059
query58	243	250	255	250
query59	3450	3224	3244	3224
query60	299	272	279	272
query61	97	93	93	93
query62	806	628	657	628
query63	312	292	300	292
query64	9174	2249	1712	1712
query65	3152	3102	3136	3102
query66	736	332	328	328
query67	15470	15049	14886	14886
query68	6113	526	544	526
query69	688	425	370	370
query70	1150	1153	1138	1138
query71	465	274	287	274
query72	7331	5660	5496	5496
query73	815	331	327	327
query74	6088	5558	5502	5502
query75	3773	2669	2707	2669
query76	3921	943	986	943
query77	664	306	322	306
query78	10366	8992	8987	8987
query79	3201	516	522	516
query80	1962	464	466	464
query81	573	219	225	219
query82	756	137	135	135
query83	264	170	167	167
query84	274	91	87	87
query85	1288	315	291	291
query86	457	331	307	307
query87	3385	3111	3125	3111
query88	3670	2453	2444	2444
query89	477	398	384	384
query90	1754	189	186	186
query91	131	102	102	102
query92	58	49	54	49
query93	4522	508	503	503
query94	1033	215	214	214
query95	399	320	318	318
query96	612	272	275	272
query97	3194	3027	3044	3027
query98	216	209	191	191
query99	1650	1234	1301	1234
Total cold run time: 286882 ms
Total hot run time: 175147 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.86 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7efd9c519cb039fa6957b752775589cdf288df11, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.03
query3	0.22	0.04	0.04
query4	1.69	0.07	0.08
query5	0.49	0.48	0.47
query6	1.13	0.73	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.54	0.48	0.48
query10	0.54	0.55	0.54
query11	0.15	0.11	0.12
query12	0.14	0.12	0.12
query13	0.59	0.60	0.59
query14	0.75	0.78	0.83
query15	0.85	0.81	0.82
query16	0.38	0.38	0.38
query17	1.06	1.04	1.04
query18	0.22	0.22	0.22
query19	1.82	1.78	1.72
query20	0.01	0.01	0.01
query21	15.39	0.74	0.66
query22	4.51	7.98	1.24
query23	18.31	1.38	1.16
query24	2.08	0.25	0.22
query25	0.16	0.08	0.08
query26	0.30	0.21	0.20
query27	0.44	0.23	0.24
query28	13.23	1.02	1.00
query29	12.58	3.30	3.27
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.47	0.49
query33	2.94	2.90	2.90
query34	17.08	4.38	4.34
query35	4.39	4.46	4.39
query36	0.66	0.47	0.48
query37	0.21	0.15	0.15
query38	0.16	0.14	0.15
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.98 s
Total hot run time: 29.86 s

@wm1581066 wm1581066 requested a review from morningman July 15, 2024 00:09
@wm1581066 wm1581066 added dev/2.1.x usercase Important user case type label p0_c labels Jul 15, 2024
@wsjz
Copy link
Contributor Author

wsjz commented Jul 15, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39988 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 409572a58f14ee30316f7d4d7d127b3a1fa9e39e, data reload: false

------ Round 1 ----------------------------------
q1	17612	4369	4245	4245
q2	2023	191	185	185
q3	10456	1146	1083	1083
q4	10192	752	787	752
q5	7556	2710	2693	2693
q6	226	137	139	137
q7	964	595	606	595
q8	9242	2061	2103	2061
q9	9063	6625	6623	6623
q10	8754	3743	3737	3737
q11	476	240	238	238
q12	480	226	230	226
q13	17887	3005	3002	3002
q14	273	232	239	232
q15	524	478	476	476
q16	510	387	383	383
q17	966	659	620	620
q18	8027	7506	7529	7506
q19	6983	1325	1331	1325
q20	690	320	305	305
q21	4961	3299	3281	3281
q22	340	283	285	283
Total cold run time: 118205 ms
Total hot run time: 39988 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4374	4234	4215	4215
q2	363	273	284	273
q3	2977	2900	2979	2900
q4	1969	1673	1729	1673
q5	5597	5529	5472	5472
q6	218	144	132	132
q7	2190	1872	1875	1872
q8	3268	3416	3421	3416
q9	8778	8847	8814	8814
q10	4154	3730	3836	3730
q11	587	508	493	493
q12	811	654	635	635
q13	16986	3164	3242	3164
q14	314	302	290	290
q15	539	483	488	483
q16	517	438	439	438
q17	1822	1531	1502	1502
q18	8116	8068	7867	7867
q19	1721	1487	1496	1487
q20	2760	1916	1861	1861
q21	9617	4865	4663	4663
q22	640	503	503	503
Total cold run time: 78318 ms
Total hot run time: 55883 ms

@wsjz
Copy link
Contributor Author

wsjz commented Jul 16, 2024

run feut

@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f7e8b0855e8ccd67d015fc43839fd4cf6a26eefd, data reload: false

------ Round 1 ----------------------------------
q1	17598	4302	4233	4233
q2	2019	189	183	183
q3	10458	1149	1145	1145
q4	10195	747	780	747
q5	7562	2684	2658	2658
q6	216	135	135	135
q7	962	592	584	584
q8	9225	2077	2079	2077
q9	8703	6528	6581	6528
q10	8738	3767	3783	3767
q11	457	238	238	238
q12	488	227	231	227
q13	17780	3004	3021	3004
q14	270	240	236	236
q15	525	473	484	473
q16	491	381	393	381
q17	955	661	739	661
q18	8056	7407	7452	7407
q19	3080	1334	1418	1334
q20	684	335	321	321
q21	5066	3227	3209	3209
q22	341	292	289	289
Total cold run time: 113869 ms
Total hot run time: 39837 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4420	4295	4224	4224
q2	388	259	278	259
q3	3001	2793	2756	2756
q4	2023	1743	1726	1726
q5	5535	5634	5525	5525
q6	215	136	152	136
q7	2153	1786	1809	1786
q8	3275	3406	3357	3357
q9	8754	8713	8768	8713
q10	4183	3877	3922	3877
q11	573	489	490	489
q12	794	636	636	636
q13	17207	3204	3212	3204
q14	322	280	299	280
q15	543	493	511	493
q16	509	430	441	430
q17	1801	1512	1513	1512
q18	7987	7902	7923	7902
q19	3685	1578	1484	1484
q20	2243	1882	1838	1838
q21	5074	4846	4860	4846
q22	571	503	523	503
Total cold run time: 75256 ms
Total hot run time: 55976 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174314 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f7e8b0855e8ccd67d015fc43839fd4cf6a26eefd, data reload: false

query1	921	362	355	355
query2	6449	1905	1927	1905
query3	6639	204	213	204
query4	28537	17647	17352	17352
query5	3733	485	474	474
query6	282	159	154	154
query7	4578	294	299	294
query8	240	202	194	194
query9	8592	2430	2399	2399
query10	425	284	279	279
query11	10885	10053	10114	10053
query12	119	86	85	85
query13	1647	396	390	390
query14	10300	7753	8055	7753
query15	248	167	168	167
query16	7884	331	329	329
query17	1780	569	555	555
query18	2003	291	282	282
query19	201	154	154	154
query20	91	84	84	84
query21	205	128	128	128
query22	4228	4013	4050	4013
query23	34119	33945	33659	33659
query24	11082	2940	2980	2940
query25	642	413	434	413
query26	990	163	153	153
query27	2262	278	284	278
query28	6606	2109	2098	2098
query29	911	658	675	658
query30	261	155	154	154
query31	958	777	760	760
query32	101	56	57	56
query33	788	311	325	311
query34	979	507	488	488
query35	701	592	614	592
query36	1173	1013	982	982
query37	146	93	84	84
query38	2901	2865	2823	2823
query39	918	851	837	837
query40	212	125	120	120
query41	52	46	49	46
query42	118	101	101	101
query43	527	508	486	486
query44	1167	729	720	720
query45	194	160	162	160
query46	1083	736	769	736
query47	1869	1804	1856	1804
query48	380	290	288	288
query49	821	406	411	406
query50	782	404	394	394
query51	6887	6936	6812	6812
query52	104	95	91	91
query53	351	287	282	282
query54	916	447	451	447
query55	72	74	74	74
query56	283	294	271	271
query57	1143	1067	1039	1039
query58	261	256	252	252
query59	2915	2707	2637	2637
query60	301	270	274	270
query61	93	94	98	94
query62	799	662	631	631
query63	318	280	289	280
query64	9514	2280	1667	1667
query65	3157	3122	3128	3122
query66	806	360	329	329
query67	15801	14967	15148	14967
query68	8584	558	560	558
query69	745	417	363	363
query70	1400	1131	1104	1104
query71	523	286	283	283
query72	9129	5569	5539	5539
query73	2281	334	332	332
query74	6261	5656	5671	5656
query75	4919	2675	2648	2648
query76	5219	995	904	904
query77	767	304	296	296
query78	9794	9286	8976	8976
query79	6096	533	524	524
query80	2519	476	482	476
query81	584	220	222	220
query82	444	141	132	132
query83	286	174	168	168
query84	264	87	85	85
query85	980	350	298	298
query86	353	326	318	318
query87	3327	3077	3107	3077
query88	4486	2398	2373	2373
query89	493	385	393	385
query90	2005	199	202	199
query91	134	102	103	102
query92	63	49	53	49
query93	5290	544	538	538
query94	1374	215	213	213
query95	400	323	330	323
query96	611	280	273	273
query97	3184	3019	3060	3019
query98	222	209	190	190
query99	1523	1278	1278	1278
Total cold run time: 300711 ms
Total hot run time: 174314 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.63 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f7e8b0855e8ccd67d015fc43839fd4cf6a26eefd, data reload: false

query1	0.04	0.04	0.02
query2	0.08	0.03	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.51	0.47	0.48
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.54	0.49	0.48
query10	0.56	0.56	0.55
query11	0.15	0.12	0.11
query12	0.14	0.13	0.13
query13	0.59	0.58	0.58
query14	0.76	0.79	0.77
query15	0.84	0.83	0.83
query16	0.36	0.36	0.36
query17	0.99	1.00	1.04
query18	0.24	0.22	0.21
query19	1.81	1.69	1.78
query20	0.01	0.00	0.01
query21	15.41	0.75	0.67
query22	4.22	7.33	1.87
query23	18.31	1.42	1.31
query24	2.04	0.24	0.23
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.22
query28	13.29	1.03	1.00
query29	12.58	3.27	3.27
query30	0.24	0.06	0.05
query31	2.89	0.39	0.39
query32	3.25	0.49	0.48
query33	2.92	2.93	2.99
query34	17.04	4.32	4.35
query35	4.41	4.41	4.42
query36	0.65	0.46	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.13
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.61 s
Total hot run time: 30.63 s

@wsjz wsjz force-pushed the fix_paimon_props branch from f7e8b08 to 43b4dfa Compare July 16, 2024 05:00
@wsjz
Copy link
Contributor Author

wsjz commented Jul 16, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40158 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8cf1bab39578a09306d024a90b8486bedfaaa475, data reload: false

------ Round 1 ----------------------------------
q1	17966	4482	4415	4415
q2	2906	199	190	190
q3	10732	1186	1169	1169
q4	10597	842	831	831
q5	7619	2727	2670	2670
q6	222	138	137	137
q7	970	606	610	606
q8	9277	2117	2103	2103
q9	8604	6630	6559	6559
q10	8715	3811	3794	3794
q11	451	247	243	243
q12	407	242	221	221
q13	17751	2954	2963	2954
q14	280	229	242	229
q15	531	470	499	470
q16	477	396	370	370
q17	959	629	602	602
q18	7997	7386	7401	7386
q19	7468	1411	1443	1411
q20	705	325	318	318
q21	4863	3212	3199	3199
q22	344	287	281	281
Total cold run time: 119841 ms
Total hot run time: 40158 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4398	4279	4291	4279
q2	377	270	272	270
q3	2954	2806	2727	2727
q4	1932	1563	1583	1563
q5	5301	5296	5316	5296
q6	220	132	130	130
q7	2106	1675	1723	1675
q8	3182	3345	3317	3317
q9	8375	8351	8372	8351
q10	3917	3760	3755	3755
q11	571	490	501	490
q12	770	620	583	583
q13	16401	2965	2977	2965
q14	310	273	272	272
q15	532	470	472	470
q16	475	405	421	405
q17	1762	1473	1438	1438
q18	7660	7531	7204	7204
q19	1670	1520	1563	1520
q20	2012	1758	1776	1758
q21	4845	4678	4629	4629
q22	577	482	513	482
Total cold run time: 70347 ms
Total hot run time: 53579 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173023 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8cf1bab39578a09306d024a90b8486bedfaaa475, data reload: false

query1	917	366	363	363
query2	6473	1879	1796	1796
query3	6674	206	225	206
query4	25909	17357	17292	17292
query5	4266	506	483	483
query6	289	177	154	154
query7	4595	291	297	291
query8	235	189	195	189
query9	8509	2433	2404	2404
query10	430	287	282	282
query11	11657	10009	10036	10009
query12	143	84	84	84
query13	1643	361	371	361
query14	10180	7921	7783	7783
query15	240	173	166	166
query16	7776	320	329	320
query17	1784	569	526	526
query18	1810	286	273	273
query19	190	153	154	153
query20	95	80	84	80
query21	203	146	129	129
query22	4369	4078	3972	3972
query23	33993	33183	32979	32979
query24	12087	2851	2911	2851
query25	670	372	366	366
query26	1737	155	152	152
query27	2839	275	279	275
query28	7414	2036	2025	2025
query29	1033	642	610	610
query30	281	150	147	147
query31	981	759	751	751
query32	93	56	59	56
query33	803	299	307	299
query34	915	500	494	494
query35	673	584	549	549
query36	1081	952	946	946
query37	201	83	81	81
query38	2829	2738	2744	2738
query39	871	813	809	809
query40	293	124	119	119
query41	47	46	46	46
query42	122	99	103	99
query43	524	479	468	468
query44	1226	745	730	730
query45	194	160	165	160
query46	1081	731	725	725
query47	1906	1784	1773	1773
query48	365	295	296	295
query49	1209	447	415	415
query50	784	402	420	402
query51	6999	6856	6795	6795
query52	110	98	100	98
query53	362	295	304	295
query54	986	460	452	452
query55	77	74	74	74
query56	319	283	286	283
query57	1163	1110	1076	1076
query58	252	267	251	251
query59	2898	2746	2824	2746
query60	328	286	303	286
query61	112	116	135	116
query62	831	652	669	652
query63	339	296	300	296
query64	10543	2195	1645	1645
query65	3162	3146	3118	3118
query66	1338	337	342	337
query67	15356	14957	14981	14957
query68	4627	563	560	560
query69	484	337	336	336
query70	1194	1196	1086	1086
query71	394	290	286	286
query72	7002	5636	5533	5533
query73	758	327	326	326
query74	6000	5702	5685	5685
query75	3502	2693	2690	2690
query76	2794	985	948	948
query77	522	309	314	309
query78	9898	8922	9004	8922
query79	2438	520	527	520
query80	1856	492	478	478
query81	579	227	220	220
query82	904	141	136	136
query83	277	164	168	164
query84	278	88	89	88
query85	2093	322	303	303
query86	491	293	305	293
query87	3271	3121	3130	3121
query88	3815	2359	2380	2359
query89	476	386	386	386
query90	1840	195	201	195
query91	134	103	102	102
query92	67	50	50	50
query93	2532	520	512	512
query94	1302	221	214	214
query95	471	315	325	315
query96	603	268	278	268
query97	3229	3049	3001	3001
query98	219	205	196	196
query99	1531	1278	1268	1268
Total cold run time: 285395 ms
Total hot run time: 173023 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8cf1bab39578a09306d024a90b8486bedfaaa475, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.67	0.10	0.09
query5	0.49	0.47	0.48
query6	1.14	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.48
query10	0.55	0.54	0.54
query11	0.15	0.11	0.11
query12	0.15	0.13	0.12
query13	0.59	0.59	0.58
query14	0.77	0.77	0.80
query15	0.86	0.82	0.80
query16	0.37	0.37	0.36
query17	1.03	0.96	0.97
query18	0.25	0.22	0.21
query19	1.84	1.69	1.74
query20	0.01	0.02	0.01
query21	15.39	0.74	0.65
query22	4.75	6.84	1.98
query23	18.26	1.38	1.27
query24	2.13	0.22	0.20
query25	0.16	0.08	0.08
query26	0.29	0.23	0.21
query27	0.46	0.23	0.23
query28	13.32	1.04	1.01
query29	12.60	3.30	3.29
query30	0.26	0.05	0.06
query31	2.86	0.39	0.38
query32	3.28	0.48	0.46
query33	2.85	2.97	2.90
query34	17.00	4.41	4.35
query35	4.37	4.46	4.44
query36	0.66	0.48	0.48
query37	0.19	0.16	0.17
query38	0.17	0.15	0.15
query39	0.05	0.04	0.03
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.05	0.06	0.06
query43	0.05	0.04	0.04
Total cold run time: 110.24 s
Total hot run time: 30.67 s

morningman
morningman approved these changes Jul 16, 2024
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@morningman morningman merged commit 358765c into apache:master Jul 16, 2024
27 of 29 checks passed
@wsjz wsjz deleted the fix_paimon_props branch July 16, 2024 07:18
morningman pushed a commit to morningman/doris that referenced this pull request Jul 16, 2024
@wsjz @morningman
[fix](multi-catalog)fix paimon meta properties convert (apache#37249)
c97599a 
## Proposed changes

1. set default region to 'us-east-1' if no any region can be read
2. add meta properties convert to paimon
3. recognize ip endpoint, and return null for getRegionOfEndpoint method
4. add ut case for meta properties convert
@morningman morningman mentioned this pull request Jul 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@morningman @wsjz
[fix](multi-catalog)fix paimon meta properties convert (#37249) (#37958)
ba66ff5 
bp #37249

Co-authored-by: slothever <18522955+wsjz@users.noreply.github.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@wsjz @seawinde
[fix](multi-catalog)fix paimon meta properties convert (apache#37249)
f33c384 
## Proposed changes

1. set default region to 'us-east-1' if no any region can be read
2. add meta properties convert to paimon
3. recognize ip endpoint, and return null for getRegionOfEndpoint method
4. add ut case for meta properties convert
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@wsjz @dataroaring
[fix](multi-catalog)fix paimon meta properties convert (#37249)
b7ca571 
## Proposed changes

1. set default region to 'us-east-1' if no any region can be read
2. add meta properties convert to paimon
3. recognize ip endpoint, and return null for getRegionOfEndpoint method
4. add ut case for meta properties convert
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@wuwenchi wuwenchi wuwenchi approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged p0_c reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@wsjz @doris-robot @morningman @wuwenchi @yiguolei @wm1581066 @dataroaring