Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](load) Fix the channel leak when close wait has been cancelled #38031 #38186

Merged
merged 1 commit into from
Jul 22, 2024
Merged

[Fix](load) Fix the channel leak when close wait has been cancelled #38031 #38186

merged 1 commit into from
Jul 22, 2024

Conversation

liaoxin01
Copy link
Contributor

cherry pick from #38031

@liaoxin01
[Fix](load) Fix the channel leak when close wait has been cancelled (a…
ec305b8 
…pache#38031)

When the close_wait is called, the NodeChannel has already been marked
as cancelled, but close_wait will set _is_closed to true. When it
actually sends a cancel request to the downstream LoadChannel, it finds
that _is_closed has already been set to true, so it will not send an RPC
request, causing a LoadChannel leak.
@liaoxin01
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 50136 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ec305b8b805cb40dd58419b17f42a31851f510dd, data reload: false

------ Round 1 ----------------------------------
q1	18232	4379	4318	4318
q2	2070	152	149	149
q3	10273	1904	1927	1904
q4	10352	1260	1323	1260
q5	8596	3905	3963	3905
q6	239	148	127	127
q7	2051	1590	1634	1590
q8	9273	2717	2700	2700
q9	10980	10847	10525	10525
q10	8616	3519	3516	3516
q11	415	243	253	243
q12	472	300	310	300
q13	18345	4014	4054	4014
q14	356	337	337	337
q15	501	454	458	454
q16	667	586	582	582
q17	1107	970	956	956
q18	7236	6902	6843	6843
q19	1759	1674	1601	1601
q20	545	301	319	301
q21	4441	4106	4085	4085
q22	532	436	426	426
Total cold run time: 117058 ms
Total hot run time: 50136 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4313	4314	4291	4291
q2	314	228	225	225
q3	4206	4149	4126	4126
q4	2741	2746	2737	2737
q5	7239	7145	7111	7111
q6	235	121	122	121
q7	3262	2833	2806	2806
q8	4390	4446	4505	4446
q9	17500	16985	17009	16985
q10	4252	4276	4281	4276
q11	731	687	689	687
q12	1044	881	887	881
q13	6479	3790	3771	3771
q14	450	426	418	418
q15	511	462	454	454
q16	742	704	694	694
q17	3880	3905	3837	3837
q18	8709	8889	8903	8889
q19	1715	1723	1649	1649
q20	2383	2126	2152	2126
q21	8600	8547	8494	8494
q22	1125	982	988	982
Total cold run time: 84821 ms
Total hot run time: 80006 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8113/21465)
Line Coverage: 29.47% (66503/225691)
Region Coverage: 28.93% (34269/118443)
Branch Coverage: 24.81% (17605/70950)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ec305b8b805cb40dd58419b17f42a31851f510dd_ec305b8b805cb40dd58419b17f42a31851f510dd/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 203064 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ec305b8b805cb40dd58419b17f42a31851f510dd, data reload: false

query1	938	424	381	381
query2	7060	2779	2656	2656
query3	9809	216	218	216
query4	23741	18074	18041	18041
query5	19738	6592	6575	6575
query6	415	218	228	218
query7	5136	309	326	309
query8	392	434	396	396
query9	3049	2651	2577	2577
query10	440	312	308	308
query11	11360	10761	10687	10687
query12	129	82	76	76
query13	5593	686	695	686
query14	18982	13537	13188	13188
query15	367	247	244	244
query16	6425	294	271	271
query17	1321	1468	871	871
query18	2299	418	410	410
query19	206	154	158	154
query20	79	81	81	81
query21	188	98	93	93
query22	5236	5024	5097	5024
query23	32495	31816	31776	31776
query24	6805	6557	6521	6521
query25	538	433	435	433
query26	494	162	166	162
query27	1697	302	294	294
query28	6127	2345	2300	2300
query29	2975	2791	2693	2693
query30	244	170	172	170
query31	942	751	754	751
query32	68	64	61	61
query33	404	256	243	243
query34	857	464	486	464
query35	1144	991	998	991
query36	1149	1284	1215	1215
query37	92	62	58	58
query38	3053	2934	2920	2920
query39	1391	1329	1297	1297
query40	207	106	99	99
query41	46	43	45	43
query42	79	84	77	77
query43	764	684	691	684
query44	1137	732	714	714
query45	243	241	237	237
query46	1228	955	983	955
query47	1923	1762	1699	1699
query48	989	705	709	705
query49	617	374	372	372
query50	880	634	601	601
query51	4800	4640	4635	4635
query52	95	87	82	82
query53	441	320	322	320
query54	2657	2447	2469	2447
query55	87	80	79	79
query56	212	223	219	219
query57	1221	1088	1211	1088
query58	213	210	233	210
query59	4159	4095	3652	3652
query60	224	194	208	194
query61	101	108	101	101
query62	817	536	494	494
query63	498	343	345	343
query64	4040	1625	1525	1525
query65	3654	3589	3544	3544
query66	657	398	377	377
query67	16065	15271	14870	14870
query68	11001	652	641	641
query69	588	355	336	336
query70	2018	1371	1281	1281
query71	425	319	320	319
query72	6572	3489	3539	3489
query73	746	329	318	318
query74	6254	5810	5817	5810
query75	5559	3682	3662	3662
query76	6964	1143	1224	1143
query77	1236	265	269	265
query78	12779	12079	11997	11997
query79	9735	645	646	645
query80	790	414	406	406
query81	457	235	235	235
query82	684	102	98	98
query83	172	135	136	135
query84	254	74	70	70
query85	877	337	339	337
query86	338	303	310	303
query87	3278	3002	3065	3002
query88	4410	2292	2282	2282
query89	455	282	281	281
query90	1957	211	219	211
query91	192	137	152	137
query92	65	56	55	55
query93	5549	571	594	571
query94	738	213	222	213
query95	1102	1050	1072	1050
query96	632	326	321	321
query97	6624	6250	6398	6250
query98	203	178	171	171
query99	3062	853	876	853
Total cold run time: 327148 ms
Total hot run time: 203064 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.05 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ec305b8b805cb40dd58419b17f42a31851f510dd, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.25	0.04	0.05
query4	1.79	0.08	0.08
query5	0.54	0.52	0.52
query6	1.25	0.65	0.61
query7	0.01	0.02	0.01
query8	0.04	0.03	0.02
query9	0.52	0.48	0.48
query10	0.53	0.54	0.54
query11	0.12	0.10	0.09
query12	0.12	0.08	0.08
query13	0.62	0.62	0.61
query14	0.77	0.80	0.78
query15	0.78	0.77	0.76
query16	0.40	0.36	0.35
query17	0.98	1.02	0.99
query18	0.23	0.24	0.24
query19	1.91	1.84	1.85
query20	0.02	0.01	0.01
query21	15.49	0.56	0.54
query22	2.16	2.50	1.68
query23	17.28	1.09	0.93
query24	6.71	1.40	1.28
query25	0.37	0.14	0.05
query26	0.74	0.14	0.15
query27	0.04	0.04	0.04
query28	6.08	0.74	0.74
query29	12.69	2.19	2.38
query30	0.57	0.59	0.53
query31	2.82	0.38	0.38
query32	3.37	0.49	0.49
query33	3.09	3.05	3.06
query34	15.28	4.81	4.80
query35	4.85	4.85	4.88
query36	1.05	1.02	1.03
query37	0.06	0.05	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 103.94 s
Total hot run time: 31.05 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit ec305b8b805cb40dd58419b17f42a31851f510dd with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@dataroaring dataroaring merged commit f55dd29 into apache:branch-2.0 Jul 22, 2024
23 of 25 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@liaoxin01
[Fix](load) Fix the channel leak when close wait has been cancelled a…
687adfa 
…pache#38031 (apache#38186)

cherry pick from apache#38031
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@liaoxin01 @doris-robot @dataroaring