Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](truncate) fix tablet invert index leaky #37334

Merged
merged 3 commits into from
Jul 5, 2024
Merged

[fix](truncate) fix tablet invert index leaky #37334

merged 3 commits into from
Jul 5, 2024

Conversation

yujun777
Copy link
Collaborator

@yujun777 yujun777 commented Jul 5, 2024

FIX:

  1. fix tablet invert index leaky due to truncate fail;
  2. fix NPE due to partitions has changed during truncating;

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@yujun777
Copy link
Collaborator Author

yujun777 commented Jul 5, 2024

run buildall

@yujun777 yujun777 changed the title [fix](truncate) fix tablet invert index leaky and truncate npe [fix](truncate) fix tablet invert index leaky Jul 5, 2024
@yujun777
Copy link
Collaborator Author

yujun777 commented Jul 5, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40077 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e9f82a9bee1070224613e2df8f7790484e07dc5e, data reload: false

------ Round 1 ----------------------------------
q1	17628	4415	4309	4309
q2	2024	193	192	192
q3	10445	1261	1074	1074
q4	10187	844	798	798
q5	7513	2692	2658	2658
q6	231	135	136	135
q7	958	596	608	596
q8	9254	2118	2092	2092
q9	9165	6563	6552	6552
q10	9018	3731	3751	3731
q11	450	239	245	239
q12	426	239	225	225
q13	17763	2979	2976	2976
q14	280	225	228	225
q15	532	494	490	490
q16	516	379	377	377
q17	985	665	649	649
q18	8135	7537	7463	7463
q19	1669	1554	1490	1490
q20	670	333	332	332
q21	5041	3136	3316	3136
q22	413	338	340	338
Total cold run time: 113303 ms
Total hot run time: 40077 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4337	4240	4580	4240
q2	373	260	272	260
q3	3045	2742	2678	2678
q4	1895	1587	1610	1587
q5	5280	5279	5265	5265
q6	217	131	130	130
q7	2080	1700	1704	1700
q8	3257	3410	3322	3322
q9	8424	8397	8395	8395
q10	3873	3697	3634	3634
q11	601	485	479	479
q12	784	604	613	604
q13	16540	2982	2947	2947
q14	298	270	261	261
q15	521	482	471	471
q16	488	422	451	422
q17	1810	1501	1473	1473
q18	7490	7506	7355	7355
q19	1725	1672	1591	1591
q20	1993	1808	1794	1794
q21	4869	4696	4789	4696
q22	638	535	538	535
Total cold run time: 70538 ms
Total hot run time: 53839 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173652 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e9f82a9bee1070224613e2df8f7790484e07dc5e, data reload: false

query1	941	383	368	368
query2	6472	2523	2387	2387
query3	6653	209	219	209
query4	28488	17455	17445	17445
query5	4173	478	489	478
query6	274	179	178	178
query7	4610	296	297	296
query8	329	298	288	288
query9	8408	2343	2323	2323
query10	626	293	269	269
query11	10749	10236	10393	10236
query12	136	82	81	81
query13	1625	369	359	359
query14	9387	7745	7576	7576
query15	228	183	188	183
query16	7830	324	302	302
query17	1814	554	522	522
query18	1954	282	283	282
query19	199	144	160	144
query20	99	83	85	83
query21	206	133	131	131
query22	4440	4255	4224	4224
query23	33644	33099	33180	33099
query24	12179	2855	2874	2855
query25	655	362	373	362
query26	1789	160	159	159
query27	3002	317	324	317
query28	7679	2020	2019	2019
query29	1123	642	620	620
query30	295	152	148	148
query31	964	733	745	733
query32	94	54	53	53
query33	777	304	308	304
query34	970	486	486	486
query35	781	628	604	604
query36	1090	952	941	941
query37	289	75	72	72
query38	2871	2751	2726	2726
query39	903	799	811	799
query40	285	126	124	124
query41	53	51	52	51
query42	118	98	99	98
query43	590	545	530	530
query44	1259	723	725	723
query45	191	163	164	163
query46	1100	712	724	712
query47	1890	1816	1812	1812
query48	383	297	301	297
query49	1207	414	410	410
query50	780	386	388	386
query51	6869	6821	6814	6814
query52	112	88	97	88
query53	359	284	286	284
query54	1033	447	441	441
query55	76	73	73	73
query56	283	276	314	276
query57	1130	1040	1052	1040
query58	255	252	253	252
query59	3278	3202	3119	3119
query60	298	308	294	294
query61	95	92	98	92
query62	660	454	450	450
query63	325	293	293	293
query64	9904	2282	1763	1763
query65	3194	3126	3144	3126
query66	1381	341	346	341
query67	15479	15103	15104	15103
query68	4502	535	530	530
query69	469	331	325	325
query70	1198	1031	1096	1031
query71	406	283	276	276
query72	7187	5177	5313	5177
query73	751	325	324	324
query74	5986	5558	5520	5520
query75	3404	2677	2748	2677
query76	2739	936	947	936
query77	466	302	307	302
query78	9410	9089	8968	8968
query79	2639	515	506	506
query80	1993	480	470	470
query81	598	218	221	218
query82	1111	109	114	109
query83	331	169	172	169
query84	285	90	91	90
query85	1607	322	296	296
query86	477	305	301	301
query87	3326	3071	3147	3071
query88	4425	2473	2447	2447
query89	477	377	377	377
query90	1831	187	190	187
query91	133	107	112	107
query92	62	50	53	50
query93	2559	495	493	493
query94	1286	219	284	219
query95	409	310	312	310
query96	596	272	268	268
query97	3210	3070	3041	3041
query98	221	201	207	201
query99	1288	854	840	840
Total cold run time: 286715 ms
Total hot run time: 173652 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.2 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e9f82a9bee1070224613e2df8f7790484e07dc5e, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.03	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.51	0.50	0.48
query6	1.14	0.72	0.72
query7	0.01	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.51
query10	0.57	0.56	0.57
query11	0.16	0.13	0.12
query12	0.14	0.12	0.13
query13	0.59	0.58	0.59
query14	0.79	0.77	0.78
query15	0.86	0.81	0.82
query16	0.35	0.35	0.36
query17	1.05	1.02	1.04
query18	0.22	0.25	0.23
query19	1.93	1.72	1.77
query20	0.02	0.01	0.01
query21	15.40	0.77	0.67
query22	4.34	7.54	1.55
query23	18.25	1.34	1.20
query24	2.07	0.22	0.22
query25	0.15	0.10	0.09
query26	0.29	0.20	0.19
query27	0.46	0.23	0.22
query28	13.32	1.02	1.00
query29	12.59	3.32	3.27
query30	0.25	0.07	0.05
query31	2.87	0.39	0.39
query32	3.27	0.47	0.47
query33	3.01	2.92	2.95
query34	16.88	4.39	4.35
query35	4.44	4.42	4.38
query36	0.64	0.46	0.47
query37	0.18	0.16	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.14	0.12
query41	0.10	0.04	0.05
query42	0.05	0.06	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.93 s
Total hot run time: 30.2 s

@yujun777
Copy link
Collaborator Author

yujun777 commented Jul 5, 2024

run external

dataroaring
dataroaring approved these changes Jul 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

deardeng
deardeng approved these changes Jul 5, 2024
View reviewed changes
Copy link
Contributor

@deardeng deardeng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 6c7defd into apache:master Jul 5, 2024
26 of 29 checks passed
This was referenced Jul 8, 2024
Merged
Merged
yiguolei pushed a commit that referenced this pull request Jul 8, 2024
@yujun777
[fix](truncate) fix tablet invert index leaky #37334 (#37410)
a6ec78e 
cherry pick from #37334
dataroaring pushed a commit that referenced this pull request Jul 8, 2024
@yujun777
[fix](truncate) fix tablet invert index leaky #37334 (#37411)
17e5177 
cherry pick from #37334
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@yujun777 @dataroaring
[fix](truncate) fix tablet invert index leaky (#37334)
78d8380 
FIX:
1. fix tablet invert index leaky due to truncate fail;
2. fix NPE due to partitions has changed during truncating;
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@yujun777
[fix](truncate) fix tablet invert index leaky apache#37334 (apache#37411
b68c37c 
)

cherry pick from apache#37334
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@deardeng deardeng deardeng approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@yujun777 @doris-robot @deardeng @dataroaring