Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[pipeline](fix) Set upstream operators always runnable once source operator closed #37297

Merged
merged 2 commits into from
Jul 4, 2024
Merged

[pipeline](fix) Set upstream operators always runnable once source operator closed #37297

merged 2 commits into from
Jul 4, 2024

Conversation

Gabriel39
Copy link
Contributor

@Gabriel39 Gabriel39 commented Jul 4, 2024 •
edited
Loading

Proposed changes

Some kinds of source operators has a 1-1 relationship with a sink operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator will not be blocked if AnalyticSourceOperator already closed.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@wm1581066 wm1581066 added usercase Important user case type label dev/2.1.x labels Jul 4, 2024
@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 40424 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4a6d896956e578d57fcc107d3c459882e2344b4b, data reload: false

------ Round 1 ----------------------------------
q1	18001	4348	4343	4343
q2	2022	196	189	189
q3	10450	1127	1070	1070
q4	10756	854	827	827
q5	8166	2693	2622	2622
q6	220	139	135	135
q7	958	593	590	590
q8	9224	2098	2089	2089
q9	8888	6639	6664	6639
q10	9236	3854	3854	3854
q11	461	250	263	250
q12	487	240	245	240
q13	18517	3100	3104	3100
q14	276	239	219	219
q15	540	510	512	510
q16	1019	390	389	389
q17	1068	592	722	592
q18	9625	7535	7384	7384
q19	8034	1596	1588	1588
q20	656	318	316	316
q21	4847	3141	3177	3141
q22	380	337	347	337
Total cold run time: 123831 ms
Total hot run time: 40424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4384	4269	4388	4269
q2	385	273	270	270
q3	3183	2911	2943	2911
q4	1989	1680	1721	1680
q5	5448	5385	5547	5385
q6	226	143	134	134
q7	2201	1841	1849	1841
q8	3284	3417	3421	3417
q9	8701	8694	8735	8694
q10	3947	3852	3758	3758
q11	581	490	512	490
q12	807	638	622	622
q13	15966	3225	3176	3176
q14	308	271	271	271
q15	534	474	480	474
q16	476	429	427	427
q17	1844	1535	1533	1533
q18	8229	7967	7779	7779
q19	1828	1564	1551	1551
q20	2193	1898	1865	1865
q21	5027	4680	4948	4680
q22	643	534	555	534
Total cold run time: 72184 ms
Total hot run time: 55761 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 171256 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4a6d896956e578d57fcc107d3c459882e2344b4b, data reload: false

query1	925	393	367	367
query2	6377	2406	2424	2406
query3	6629	200	221	200
query4	19466	17545	17108	17108
query5	3631	473	466	466
query6	276	169	164	164
query7	4595	314	298	298
query8	308	279	284	279
query9	8373	2405	2389	2389
query10	584	285	283	283
query11	10619	9975	10016	9975
query12	116	84	79	79
query13	1635	368	359	359
query14	9939	6059	6708	6059
query15	229	183	185	183
query16	7389	315	304	304
query17	1378	537	519	519
query18	1929	273	297	273
query19	195	151	146	146
query20	89	81	80	80
query21	213	128	129	128
query22	4537	4090	4030	4030
query23	33642	33549	33321	33321
query24	11117	2863	2894	2863
query25	653	387	393	387
query26	1195	157	155	155
query27	3105	320	330	320
query28	7374	2142	2135	2135
query29	911	645	643	643
query30	257	155	154	154
query31	1011	739	739	739
query32	99	52	55	52
query33	787	294	287	287
query34	1029	472	485	472
query35	748	635	646	635
query36	1126	968	931	931
query37	151	86	79	79
query38	2980	2786	2827	2786
query39	911	833	826	826
query40	216	130	132	130
query41	56	49	50	49
query42	113	97	104	97
query43	594	567	548	548
query44	1204	743	722	722
query45	200	170	161	161
query46	1078	705	749	705
query47	1847	1797	1800	1797
query48	378	298	292	292
query49	863	402	417	402
query50	781	376	383	376
query51	6757	6750	6704	6704
query52	98	94	94	94
query53	355	301	283	283
query54	840	438	446	438
query55	74	75	73	73
query56	293	266	269	266
query57	1123	1034	1034	1034
query58	250	246	297	246
query59	3377	3034	3143	3034
query60	310	290	285	285
query61	99	95	97	95
query62	613	442	449	442
query63	321	285	288	285
query64	9030	2247	1765	1765
query65	3139	3086	3438	3086
query66	1327	322	333	322
query67	15347	14953	14904	14904
query68	4483	525	538	525
query69	458	300	299	299
query70	1116	1116	1139	1116
query71	401	275	284	275
query72	7251	5620	5589	5589
query73	743	323	322	322
query74	5872	5583	5446	5446
query75	3426	2628	2670	2628
query76	2282	914	837	837
query77	452	309	307	307
query78	9521	9047	8842	8842
query79	2340	513	522	513
query80	2215	466	466	466
query81	589	221	222	221
query82	836	106	100	100
query83	301	172	175	172
query84	261	87	90	87
query85	2076	290	275	275
query86	488	333	316	316
query87	3257	3056	3078	3056
query88	3853	2375	2352	2352
query89	466	366	368	366
query90	1773	186	186	186
query91	132	101	104	101
query92	65	49	48	48
query93	2383	498	502	498
query94	1154	209	218	209
query95	398	310	315	310
query96	584	266	268	266
query97	3179	2981	3041	2981
query98	210	203	198	198
query99	1186	863	834	834
Total cold run time: 270962 ms
Total hot run time: 171256 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4a6d896956e578d57fcc107d3c459882e2344b4b, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.67	0.08	0.08
query5	0.50	0.47	0.47
query6	1.15	0.72	0.72
query7	0.02	0.01	0.01
query8	0.06	0.04	0.05
query9	0.56	0.49	0.50
query10	0.54	0.55	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.11
query13	0.60	0.59	0.58
query14	0.79	0.79	0.77
query15	0.84	0.80	0.81
query16	0.36	0.36	0.36
query17	1.00	0.95	0.95
query18	0.23	0.25	0.24
query19	1.81	1.65	1.74
query20	0.01	0.00	0.01
query21	15.38	0.76	0.65
query22	4.46	7.31	1.62
query23	18.30	1.43	1.33
query24	2.14	0.22	0.22
query25	0.16	0.08	0.09
query26	0.29	0.21	0.22
query27	0.45	0.23	0.24
query28	13.18	1.00	1.01
query29	12.67	3.30	3.30
query30	0.25	0.07	0.05
query31	2.86	0.40	0.38
query32	3.26	0.48	0.46
query33	2.89	2.90	2.86
query34	17.02	4.29	4.36
query35	4.38	4.44	4.39
query36	0.64	0.46	0.46
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.04	0.03	0.04
query40	0.14	0.12	0.13
query41	0.09	0.04	0.05
query42	0.06	0.06	0.04
query43	0.05	0.04	0.04
Total cold run time: 109.84 s
Total hot run time: 30.14 s

@Gabriel39 Gabriel39 merged commit b1cd368 into apache:master Jul 4, 2024
26 of 29 checks passed
Gabriel39 added a commit to Gabriel39/incubator-doris that referenced this pull request Jul 5, 2024
@Gabriel39
[pipeline](fix) Set upstream operators always runnable once source op…
2676081 
…erator closed (apache#37297)

Some kinds of source operators has a 1-1 relationship with a sink
operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator
will not be blocked if AnalyticSourceOperator already closed.
@Gabriel39 Gabriel39 mentioned this pull request Jul 5, 2024
Merged
Gabriel39 added a commit that referenced this pull request Jul 5, 2024
@Gabriel39
[pipeline](fix) Set upstream operators always runnable once source op… (
a803e14 
#37325)

…erator closed (#37297)

Some kinds of source operators has a 1-1 relationship with a sink
operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator
will not be blocked if AnalyticSourceOperator already closed.

pick #37297
dataroaring pushed a commit that referenced this pull request Jul 9, 2024
@Gabriel39 @dataroaring
[pipeline](fix) Set upstream operators always runnable once source op…
5f44a77 
…erator closed (#37297)

Some kinds of source operators has a 1-1 relationship with a sink
operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator
will not be blocked if AnalyticSourceOperator already closed.
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@Gabriel39 @dataroaring
[pipeline](fix) Set upstream operators always runnable once source op…
63c68ce 
…erator closed (#37297)

Some kinds of source operators has a 1-1 relationship with a sink
operator (such as AnalyticOperator). We must ensure AnalyticSinkOperator
will not be blocked if AnalyticSourceOperator already closed.
@yiguolei yiguolei mentioned this pull request Jul 19, 2024
Open
1 task
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangstar333 zhangstar333 zhangstar333 approved these changes

@Mryange Mryange Mryange approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed usercase Important user case type label
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Gabriel39 @doris-robot @Mryange @zhangstar333 @wm1581066 @dataroaring