Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(cloud-merge) Fix index data cannot be writed into ttl dir #37485

Merged
merged 2 commits into from
Jul 18, 2024

Conversation

Lchangliang
Copy link
Contributor

When write index data in ttl dir, it will cannot find the dir because expiration_time is 0. Reset context.expiration_time if the file is ttl file.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Jul 8, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39757 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 51701da358ec5833169e2bdb0169593b34a73523, data reload: false

------ Round 1 ----------------------------------
q1	17644	4404	4237	4237
q2	2025	192	186	186
q3	10520	1209	1106	1106
q4	10201	850	706	706
q5	7509	2667	2818	2667
q6	229	139	136	136
q7	963	595	608	595
q8	9252	2128	2111	2111
q9	8958	6552	6536	6536
q10	9001	3762	3690	3690
q11	445	240	242	240
q12	468	229	230	229
q13	17776	3005	2970	2970
q14	274	236	229	229
q15	521	471	484	471
q16	490	370	376	370
q17	978	748	632	632
q18	8137	7489	7403	7403
q19	4448	1538	1440	1440
q20	682	331	349	331
q21	4959	3130	4052	3130
q22	388	348	342	342
Total cold run time: 115868 ms
Total hot run time: 39757 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4363	4306	4259	4259
q2	362	263	256	256
q3	2997	2808	2950	2808
q4	2047	1748	1723	1723
q5	5690	5570	5534	5534
q6	220	129	131	129
q7	2171	1853	1917	1853
q8	3270	3433	3458	3433
q9	8749	8720	8861	8720
q10	4080	3912	3677	3677
q11	620	509	511	509
q12	819	668	660	660
q13	16133	3208	3161	3161
q14	316	277	278	277
q15	539	480	494	480
q16	482	427	444	427
q17	1841	1520	1543	1520
q18	8120	8052	7840	7840
q19	1766	1795	1619	1619
q20	2057	1884	1854	1854
q21	5147	4969	4766	4766
q22	635	558	546	546
Total cold run time: 72424 ms
Total hot run time: 56051 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174198 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 51701da358ec5833169e2bdb0169593b34a73523, data reload: false

query1	912	381	369	369
query2	6453	2291	2201	2201
query3	6638	205	218	205
query4	25273	17516	17220	17220
query5	3673	484	477	477
query6	247	161	169	161
query7	4661	307	286	286
query8	328	295	289	289
query9	8602	2408	2391	2391
query10	572	292	287	287
query11	12234	10018	9951	9951
query12	129	84	82	82
query13	1644	386	378	378
query14	10567	7454	7627	7454
query15	240	208	185	185
query16	7800	315	318	315
query17	1784	567	534	534
query18	1991	289	279	279
query19	200	154	157	154
query20	91	85	84	84
query21	209	131	127	127
query22	4384	4101	4025	4025
query23	34179	33687	33520	33520
query24	11624	2990	2971	2971
query25	687	400	420	400
query26	1727	159	156	156
query27	2947	279	291	279
query28	7625	2185	2149	2149
query29	1106	679	681	679
query30	253	160	153	153
query31	956	779	782	779
query32	107	66	62	62
query33	761	309	320	309
query34	974	506	495	495
query35	706	611	601	601
query36	1130	988	983	983
query37	291	90	89	89
query38	2990	2823	2781	2781
query39	898	909	822	822
query40	257	126	127	126
query41	52	49	52	49
query42	118	103	103	103
query43	590	545	533	533
query44	1241	766	748	748
query45	205	165	161	161
query46	1082	735	709	709
query47	1842	1773	1779	1773
query48	399	310	305	305
query49	1022	409	411	409
query50	781	391	395	391
query51	6815	6905	6842	6842
query52	105	92	92	92
query53	357	292	291	291
query54	917	451	445	445
query55	73	81	75	75
query56	298	267	272	267
query57	1124	1070	1067	1067
query58	258	238	244	238
query59	3358	3237	3171	3171
query60	301	279	300	279
query61	119	93	96	93
query62	793	645	648	645
query63	316	287	288	287
query64	10272	2202	1618	1618
query65	3169	3120	3123	3120
query66	1246	331	326	326
query67	15794	15008	14966	14966
query68	8509	562	545	545
query69	732	438	362	362
query70	1240	1157	1111	1111
query71	544	281	285	281
query72	8960	5621	5323	5323
query73	1468	322	332	322
query74	5913	5548	5515	5515
query75	5362	2652	2699	2652
query76	5052	942	974	942
query77	768	298	303	298
query78	10289	9083	8795	8795
query79	9645	527	515	515
query80	1090	528	476	476
query81	575	218	221	218
query82	723	134	139	134
query83	341	165	165	165
query84	273	84	87	84
query85	1354	316	298	298
query86	411	315	338	315
query87	3385	3106	3092	3092
query88	4047	2459	2443	2443
query89	514	385	427	385
query90	2107	189	184	184
query91	133	100	103	100
query92	66	49	47	47
query93	6900	520	518	518
query94	1345	209	205	205
query95	407	318	316	316
query96	624	280	275	275
query97	3190	3023	3012	3012
query98	217	196	195	195
query99	1570	1264	1273	1264
Total cold run time: 308251 ms
Total hot run time: 174198 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.39 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 51701da358ec5833169e2bdb0169593b34a73523, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.50	0.47	0.49
query6	1.15	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.56	0.54	0.55
query11	0.16	0.11	0.11
query12	0.15	0.11	0.12
query13	0.59	0.58	0.57
query14	0.77	0.78	0.77
query15	0.83	0.81	0.82
query16	0.36	0.35	0.35
query17	1.04	1.03	0.97
query18	0.23	0.22	0.21
query19	1.79	1.77	1.69
query20	0.01	0.02	0.01
query21	15.39	0.77	0.67
query22	4.36	7.36	1.88
query23	18.33	1.39	1.25
query24	2.07	0.22	0.22
query25	0.16	0.10	0.08
query26	0.28	0.20	0.21
query27	0.45	0.23	0.22
query28	13.25	1.01	1.00
query29	12.61	3.25	3.25
query30	0.25	0.06	0.06
query31	2.87	0.38	0.41
query32	3.26	0.49	0.46
query33	2.87	2.90	2.93
query34	16.96	4.34	4.37
query35	4.52	4.45	4.40
query36	0.64	0.45	0.49
query37	0.20	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.04	0.06
query43	0.05	0.04	0.03
Total cold run time: 109.79 s
Total hot run time: 30.39 s

@Lchangliang
Copy link
Contributor Author

run buildall

Copy link
Contributor

github-actions bot commented Jul 9, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40139 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e32ca88238c0105f65f896f5edfbeb259539fc4a, data reload: false

------ Round 1 ----------------------------------
q1	17598	4350	4327	4327
q2	2017	192	185	185
q3	10458	1164	1168	1164
q4	10187	801	797	797
q5	7515	2696	2616	2616
q6	220	141	141	141
q7	953	609	595	595
q8	9232	2073	2064	2064
q9	8934	6530	6566	6530
q10	9056	3737	3740	3737
q11	468	237	235	235
q12	458	228	230	228
q13	17769	2991	2981	2981
q14	269	213	221	213
q15	525	472	487	472
q16	508	380	371	371
q17	968	765	734	734
q18	7986	7468	7447	7447
q19	6539	1552	1452	1452
q20	653	318	329	318
q21	4996	3197	3937	3197
q22	397	335	335	335
Total cold run time: 117706 ms
Total hot run time: 40139 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4474	4270	4300	4270
q2	375	279	262	262
q3	2947	2900	2904	2900
q4	1979	1640	1765	1640
q5	5593	5493	5469	5469
q6	222	141	139	139
q7	2209	1895	1877	1877
q8	3251	3432	3456	3432
q9	8655	8702	8751	8702
q10	4105	3798	3781	3781
q11	587	518	495	495
q12	813	636	621	621
q13	17180	3152	3173	3152
q14	308	279	269	269
q15	540	478	480	478
q16	493	421	438	421
q17	1827	1538	1513	1513
q18	8208	8079	7863	7863
q19	1791	1745	1642	1642
q20	2608	1871	1877	1871
q21	6498	4880	5066	4880
q22	629	550	548	548
Total cold run time: 75292 ms
Total hot run time: 56225 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173634 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e32ca88238c0105f65f896f5edfbeb259539fc4a, data reload: false

query1	909	370	371	370
query2	6386	2328	2268	2268
query3	6674	210	219	210
query4	27849	17310	17114	17114
query5	3789	488	473	473
query6	258	163	159	159
query7	4582	290	283	283
query8	301	288	273	273
query9	8535	2466	2429	2429
query10	447	288	268	268
query11	11322	9999	10118	9999
query12	126	85	86	85
query13	1655	379	373	373
query14	9726	7140	7820	7140
query15	237	184	186	184
query16	7423	323	315	315
query17	1698	575	545	545
query18	1878	287	285	285
query19	207	158	163	158
query20	94	84	90	84
query21	204	127	131	127
query22	4436	4217	3995	3995
query23	33961	33686	33338	33338
query24	11016	2923	2874	2874
query25	607	426	453	426
query26	714	166	157	157
query27	2303	280	294	280
query28	6190	2200	2192	2192
query29	957	645	635	635
query30	254	156	159	156
query31	950	760	773	760
query32	105	65	63	63
query33	711	312	288	288
query34	909	497	496	496
query35	714	561	575	561
query36	1138	1017	951	951
query37	140	86	84	84
query38	3014	2814	2803	2803
query39	904	828	790	790
query40	205	123	123	123
query41	53	50	51	50
query42	118	104	98	98
query43	558	513	544	513
query44	1175	754	742	742
query45	193	156	160	156
query46	1093	724	743	724
query47	1836	1784	1745	1745
query48	366	298	304	298
query49	832	405	428	405
query50	788	390	391	390
query51	6922	6868	6779	6779
query52	104	90	93	90
query53	365	293	294	293
query54	915	453	448	448
query55	75	72	74	72
query56	277	259	263	259
query57	1157	1043	1052	1043
query58	276	250	265	250
query59	3316	3058	3140	3058
query60	298	274	273	273
query61	118	94	98	94
query62	805	646	651	646
query63	334	285	288	285
query64	9534	2156	1620	1620
query65	3189	3103	3127	3103
query66	791	326	327	326
query67	15694	15160	14829	14829
query68	4603	548	546	546
query69	694	421	347	347
query70	1207	1122	1065	1065
query71	461	281	284	281
query72	8406	5573	5605	5573
query73	755	330	324	324
query74	5911	5778	5850	5778
query75	4339	2689	2703	2689
query76	3066	968	887	887
query77	676	312	319	312
query78	9671	8932	8925	8925
query79	3737	524	541	524
query80	1735	481	522	481
query81	588	219	223	219
query82	1040	139	142	139
query83	353	180	174	174
query84	278	86	98	86
query85	1761	314	309	309
query86	455	310	327	310
query87	3305	3110	3072	3072
query88	4693	2383	2376	2376
query89	491	378	380	378
query90	1935	190	194	190
query91	138	105	112	105
query92	63	49	51	49
query93	5237	531	516	516
query94	1320	210	224	210
query95	414	314	322	314
query96	600	275	270	270
query97	3252	3065	3017	3017
query98	220	204	198	198
query99	1537	1299	1267	1267
Total cold run time: 287651 ms
Total hot run time: 173634 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.96 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e32ca88238c0105f65f896f5edfbeb259539fc4a, data reload: false

query1	0.04	0.04	0.03
query2	0.09	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.07
query5	0.50	0.48	0.49
query6	1.15	0.73	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.49
query10	0.54	0.55	0.56
query11	0.15	0.12	0.12
query12	0.16	0.12	0.13
query13	0.60	0.59	0.59
query14	0.76	0.77	0.77
query15	0.86	0.82	0.82
query16	0.37	0.37	0.35
query17	0.95	0.96	0.94
query18	0.23	0.22	0.22
query19	1.89	1.83	1.86
query20	0.01	0.01	0.02
query21	15.39	0.82	0.66
query22	4.08	7.62	2.13
query23	18.25	1.35	1.26
query24	2.10	0.26	0.23
query25	0.15	0.09	0.09
query26	0.30	0.22	0.21
query27	0.45	0.23	0.24
query28	13.22	1.04	1.00
query29	12.66	3.38	3.35
query30	0.26	0.06	0.06
query31	2.88	0.39	0.39
query32	3.26	0.49	0.47
query33	2.89	2.89	2.90
query34	17.06	4.39	4.31
query35	4.46	4.46	4.38
query36	0.65	0.47	0.48
query37	0.20	0.16	0.16
query38	0.17	0.15	0.15
query39	0.04	0.04	0.03
query40	0.16	0.12	0.13
query41	0.09	0.05	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.65 s
Total hot run time: 30.96 s

Copy link
Contributor

github-actions bot commented Jul 9, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 9, 2024
Copy link
Contributor

github-actions bot commented Jul 9, 2024

PR approved by anyone and no changes requested.

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 62ac255 into apache:master Jul 18, 2024
26 of 29 checks passed
dataroaring pushed a commit that referenced this pull request Jul 19, 2024
When write index data in ttl dir, it will cannot find the dir because
expiration_time is 0. Reset context.expiration_time if the file is ttl
file.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants