Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](multicatalog) make lastdbofcatalog a session variable #37826

Merged
merged 3 commits into from
Jul 23, 2024

Conversation

Yulei-Yang
Copy link
Contributor

@Yulei-Yang Yulei-Yang commented Jul 15, 2024

bp #37828

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49840 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6e237e497fdeb933cc5b3f741a29fe7cc4366119, data reload: false

------ Round 1 ----------------------------------
q1	17981	4411	4376	4376
q2	2069	158	149	149
q3	10474	1901	1900	1900
q4	10337	1232	1334	1232
q5	8408	3899	3936	3899
q6	237	126	125	125
q7	2029	1596	1609	1596
q8	9318	2715	2712	2712
q9	10526	10338	10283	10283
q10	8669	3571	3556	3556
q11	420	245	235	235
q12	469	309	312	309
q13	18337	3960	4008	3960
q14	359	336	342	336
q15	508	451	468	451
q16	672	558	567	558
q17	1130	963	952	952
q18	7383	6799	6787	6787
q19	1772	1661	1599	1599
q20	544	318	322	318
q21	4407	4140	4064	4064
q22	528	451	443	443
Total cold run time: 116577 ms
Total hot run time: 49840 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4349	4305	4318	4305
q2	322	225	230	225
q3	4196	4152	4147	4147
q4	2778	2741	2734	2734
q5	7189	7165	7121	7121
q6	234	117	117	117
q7	3202	2859	2911	2859
q8	4332	4474	4549	4474
q9	16948	16759	16717	16717
q10	4267	4256	4232	4232
q11	763	678	664	664
q12	1049	864	873	864
q13	6935	3715	3715	3715
q14	461	414	424	414
q15	505	454	455	454
q16	726	690	684	684
q17	3837	3877	3891	3877
q18	8819	8831	8630	8630
q19	1736	1707	1670	1670
q20	2377	2113	2117	2113
q21	8520	8438	8464	8438
q22	1076	1007	1006	1006
Total cold run time: 84621 ms
Total hot run time: 79460 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203964 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6e237e497fdeb933cc5b3f741a29fe7cc4366119, data reload: false

query1	925	420	381	381
query2	6536	2827	2708	2708
query3	6938	208	203	203
query4	20946	18083	18084	18083
query5	19742	6551	6530	6530
query6	398	220	227	220
query7	4999	300	302	300
query8	446	412	421	412
query9	3086	2628	2563	2563
query10	420	305	303	303
query11	11549	11074	10842	10842
query12	125	76	75	75
query13	5614	695	694	694
query14	18984	13560	13230	13230
query15	411	252	249	249
query16	6438	283	262	262
query17	1388	1490	874	874
query18	2310	412	404	404
query19	210	147	147	147
query20	80	79	80	79
query21	193	95	92	92
query22	5390	5194	5090	5090
query23	32463	31996	31818	31818
query24	6705	6509	6495	6495
query25	524	427	413	413
query26	533	165	166	165
query27	1795	289	289	289
query28	6186	2349	2311	2311
query29	2980	2771	2775	2771
query30	241	164	167	164
query31	946	760	736	736
query32	68	63	59	59
query33	390	260	252	252
query34	840	458	478	458
query35	1122	900	911	900
query36	1370	1018	1178	1018
query37	91	58	62	58
query38	3057	3020	2929	2929
query39	1399	1336	1315	1315
query40	207	97	90	90
query41	50	45	46	45
query42	83	84	84	84
query43	866	679	730	679
query44	1133	709	718	709
query45	252	245	236	236
query46	1205	957	994	957
query47	1783	1897	1668	1668
query48	1033	711	703	703
query49	629	367	366	366
query50	877	633	599	599
query51	4805	4632	4660	4632
query52	92	89	86	86
query53	448	321	323	321
query54	2614	2424	2433	2424
query55	93	79	86	79
query56	219	206	196	196
query57	1194	1155	1107	1107
query58	218	215	203	203
query59	4092	3784	4080	3784
query60	220	205	215	205
query61	97	93	94	93
query62	841	518	527	518
query63	480	335	337	335
query64	2435	1498	1511	1498
query65	4105	3553	3597	3553
query66	764	378	388	378
query67	15768	15561	16984	15561
query68	8709	656	667	656
query69	579	359	354	354
query70	1824	1402	1317	1317
query71	427	313	321	313
query72	6500	3466	3488	3466
query73	723	324	303	303
query74	6281	5867	5840	5840
query75	5260	3720	3640	3640
query76	5272	1171	1159	1159
query77	925	247	248	247
query78	12801	12151	11981	11981
query79	11861	669	647	647
query80	1641	397	392	392
query81	501	237	238	237
query82	1285	102	97	97
query83	161	130	128	128
query84	261	70	69	69
query85	909	324	337	324
query86	336	300	296	296
query87	3270	3018	3031	3018
query88	4962	2290	2295	2290
query89	427	285	273	273
query90	1992	212	212	212
query91	171	143	143	143
query92	60	53	53	53
query93	5894	569	585	569
query94	737	213	212	212
query95	1118	1052	1042	1042
query96	638	333	326	326
query97	6552	6351	6453	6351
query98	183	178	174	174
query99	3061	917	834	834
Total cold run time: 319762 ms
Total hot run time: 203964 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6e237e497fdeb933cc5b3f741a29fe7cc4366119, data reload: false

query1	0.02	0.03	0.03
query2	0.06	0.03	0.02
query3	0.25	0.04	0.05
query4	1.78	0.06	0.06
query5	0.53	0.53	0.51
query6	1.22	0.62	0.67
query7	0.01	0.01	0.01
query8	0.03	0.03	0.03
query9	0.52	0.47	0.48
query10	0.54	0.52	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.80	0.78	0.79
query15	0.79	0.76	0.76
query16	0.36	0.37	0.34
query17	1.01	1.00	1.03
query18	0.21	0.24	0.25
query19	1.94	1.86	1.85
query20	0.01	0.00	0.01
query21	15.44	0.56	0.55
query22	1.90	2.64	1.86
query23	17.16	1.08	0.94
query24	6.50	1.32	1.29
query25	0.35	0.13	0.06
query26	0.74	0.17	0.16
query27	0.03	0.04	0.04
query28	6.04	0.72	0.74
query29	12.63	2.32	2.33
query30	0.63	0.59	0.54
query31	2.81	0.40	0.38
query32	3.37	0.49	0.49
query33	3.08	3.12	3.08
query34	15.26	4.80	4.77
query35	4.86	4.81	4.82
query36	1.08	1.02	1.01
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.02
query40	0.17	0.15	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.22 s
Total hot run time: 31.33 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6e237e497fdeb933cc5b3f741a29fe7cc4366119 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

@Yulei-Yang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49480 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 107689116cb320b8172aad1d4e3712541ba8520b, data reload: false

------ Round 1 ----------------------------------
q1	17634	4359	4339	4339
q2	2078	155	147	147
q3	10462	1855	1935	1855
q4	10304	1243	1289	1243
q5	8925	3862	3867	3862
q6	258	125	125	125
q7	2028	1609	1595	1595
q8	9288	2717	2683	2683
q9	11014	10609	10219	10219
q10	8725	3486	3505	3486
q11	429	234	255	234
q12	476	303	299	299
q13	18384	3910	3984	3910
q14	360	319	322	319
q15	490	456	453	453
q16	668	578	562	562
q17	1126	968	939	939
q18	7321	6783	6970	6783
q19	1771	1632	1621	1621
q20	559	310	302	302
q21	4411	4080	4083	4080
q22	537	455	424	424
Total cold run time: 117248 ms
Total hot run time: 49480 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4322	4288	4277	4277
q2	317	227	216	216
q3	4190	4134	4105	4105
q4	2725	2707	2735	2707
q5	7169	7120	7094	7094
q6	236	116	119	116
q7	3236	2863	2929	2863
q8	4334	4426	4408	4408
q9	17403	17006	17148	17006
q10	4241	4248	4281	4248
q11	779	676	694	676
q12	1045	860	858	858
q13	7207	3715	3755	3715
q14	443	426	430	426
q15	505	467	455	455
q16	729	688	659	659
q17	3769	3793	3840	3793
q18	8844	8730	8771	8730
q19	1699	1696	1608	1608
q20	2378	2120	2149	2120
q21	8486	8463	8467	8463
q22	1036	965	1005	965
Total cold run time: 85093 ms
Total hot run time: 79508 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203463 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 107689116cb320b8172aad1d4e3712541ba8520b, data reload: false

query1	927	421	378	378
query2	6528	2726	2733	2726
query3	6941	205	205	205
query4	20229	17977	17980	17977
query5	19727	6463	6406	6406
query6	291	222	234	222
query7	4154	298	296	296
query8	394	393	418	393
query9	3015	2601	2540	2540
query10	427	320	291	291
query11	11398	10811	10700	10700
query12	118	75	79	75
query13	5602	689	685	685
query14	17710	13560	13443	13443
query15	363	233	241	233
query16	6449	284	258	258
query17	1727	1435	870	870
query18	2320	403	405	403
query19	211	146	145	145
query20	82	78	81	78
query21	196	102	97	97
query22	5356	5133	5081	5081
query23	32446	31791	31743	31743
query24	7091	6519	6561	6519
query25	524	423	426	423
query26	613	162	156	156
query27	2040	286	290	286
query28	6185	2319	2289	2289
query29	2904	2680	2811	2680
query30	242	165	163	163
query31	905	715	738	715
query32	69	60	63	60
query33	400	258	255	255
query34	857	473	472	472
query35	1114	920	902	902
query36	1479	1324	1227	1227
query37	89	59	60	59
query38	3067	2880	2915	2880
query39	1360	1325	1318	1318
query40	213	93	97	93
query41	47	42	43	42
query42	89	79	83	79
query43	705	701	729	701
query44	1153	709	708	708
query45	244	237	239	237
query46	1237	951	967	951
query47	2024	1646	1774	1646
query48	1012	707	696	696
query49	624	379	362	362
query50	861	646	641	641
query51	4804	4643	4673	4643
query52	93	76	88	76
query53	440	329	315	315
query54	2669	2440	2449	2440
query55	88	78	90	78
query56	239	203	213	203
query57	1364	1119	1017	1017
query58	209	213	201	201
query59	4027	3962	3765	3765
query60	211	192	204	192
query61	101	97	96	96
query62	816	445	467	445
query63	487	338	341	338
query64	2659	1517	1506	1506
query65	3603	3561	3536	3536
query66	843	384	377	377
query67	18092	17108	15635	15635
query68	8202	653	666	653
query69	564	350	344	344
query70	1566	1437	1351	1351
query71	404	304	315	304
query72	6501	3510	3511	3510
query73	793	326	321	321
query74	6324	5881	5793	5793
query75	4600	3747	3681	3681
query76	4591	1145	1200	1145
query77	544	258	255	255
query78	12960	11744	11806	11744
query79	9083	661	643	643
query80	1519	398	405	398
query81	499	239	231	231
query82	1689	100	94	94
query83	166	133	137	133
query84	254	71	72	71
query85	899	340	339	339
query86	330	288	283	283
query87	3362	3028	3016	3016
query88	5224	2302	2287	2287
query89	388	271	287	271
query90	1918	209	209	209
query91	183	144	140	140
query92	57	54	54	54
query93	6365	546	560	546
query94	754	204	209	204
query95	1099	1047	1051	1047
query96	640	319	325	319
query97	6547	6465	6385	6385
query98	191	176	167	167
query99	2972	857	857	857
Total cold run time: 315692 ms
Total hot run time: 203463 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.33 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 107689116cb320b8172aad1d4e3712541ba8520b, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.06	0.05
query4	1.80	0.07	0.05
query5	0.53	0.52	0.51
query6	1.32	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.03	0.02
query9	0.52	0.46	0.49
query10	0.54	0.55	0.53
query11	0.13	0.08	0.08
query12	0.11	0.09	0.09
query13	0.62	0.62	0.62
query14	0.76	0.80	0.79
query15	0.77	0.75	0.76
query16	0.36	0.37	0.37
query17	1.00	1.02	1.03
query18	0.22	0.24	0.22
query19	1.92	1.87	1.84
query20	0.01	0.01	0.01
query21	15.45	0.55	0.55
query22	1.98	2.31	1.47
query23	17.06	1.10	0.81
query24	4.26	0.88	1.99
query25	0.40	0.09	0.06
query26	0.55	0.16	0.17
query27	0.05	0.04	0.03
query28	7.99	0.72	0.78
query29	12.78	2.22	2.22
query30	0.54	0.53	0.53
query31	2.81	0.38	0.37
query32	3.39	0.50	0.49
query33	3.12	3.07	3.07
query34	15.26	4.83	4.77
query35	4.89	4.85	4.85
query36	1.05	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.06	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.94 s
Total hot run time: 30.33 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 107689116cb320b8172aad1d4e3712541ba8520b with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 23, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 17d01bb into apache:branch-2.0 Jul 23, 2024
28 of 30 checks passed
@Yulei-Yang Yulei-Yang deleted the fix_lastdbofcatalog branch July 23, 2024 03:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. kind/test reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants