Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](agg) fix RowsProduced counter is not set #38271

Merged
merged 1 commit into from
Jul 24, 2024
Merged

[fix](agg) fix RowsProduced counter is not set #38271

merged 1 commit into from
Jul 24, 2024

Conversation

jacktengg
Copy link
Contributor

@jacktengg jacktengg commented Jul 23, 2024 •
edited
Loading

Proposed changes

RowsProduced and BlocksProduced counter of agg source is not right:

Pipeline  :  0(instance_num=4):
                      RESULT_SINK_OPERATOR  (id=0):
                            -  PlanInfo
                                  -  TABLE:  rqg_1333358549.table_10_undef_partitions2_keys3_properties4_distributed_by52(table_10_undef_partitions2_keys3_properties4_distributed_by52),  PREAGGREGATION:  ON
                                  -  partitions=1/1  (table_10_undef_partitions2_keys3_properties4_distributed_by52)
                                  -  tablets=10/10,  tabletList=39981436181982,39981436181984,39981436181986  ...
                                  -  cardinality=10,  avgRowSize=0.0,  numNodes=1
                                  -  pushAggOp=NONE
                            -  BlocksProduced:  sum  4,  avg  1,  max  1,  min  1
                            -  CloseTime:  avg  28.396us,  max  43.17us,  min  19.647us
                            -  ExecTime:  avg  514.688us,  max  694.677us,  min  353.811us
                            -  InitTime:  avg  52.136us,  max  55.309us,  min  46.966us
                            -  InputRows:  sum  0,  avg  0,  max  0,  min  0
                            -  MemoryUsage:  sum  ,  avg  ,  max  ,  min  
                                -  PeakMemoryUsage:  sum  0.00  ,  avg  0.00  ,  max  0.00  ,  min  0.00  
                            -  OpenTime:  avg  212.328us,  max  249.375us,  min  170.678us
                            -  RowsProduced:  sum  8,  avg  2,  max  3,  min  0
                            -  WaitForDependencyTime:  avg  0ns,  max  0ns,  min  0ns
                            -  WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time:  avg  0ns,  max  0ns,  min  0ns
                          AGGREGATION_OPERATOR  (id=10  ,  nereids_id=598):
                                -  PlanInfo
                                      -  output:  count(pk)[#31]
                                      -  group  by:  col_varchar_10__undef_signed
                                      -  sortByGroupKey:false
                                      -  cardinality=8
                                      -  projections:  field1,  col_varchar_10__undef_signed
                                      -  project  output  tuple  id:  11
                                -  BlocksProduced:  sum  0,  avg  0,  max  0,  min  0
                                -  CloseTime:  avg  5.617us,  max  6.543us,  min  5.247us
                                -  ExecTime:  avg  1.172ms,  max  1.609ms,  min  289.815us
                                -  InitTime:  avg  0ns,  max  0ns,  min  0ns
                                -  MemoryUsage:  sum  ,  avg  ,  max  ,  min  
                                    -  PeakMemoryUsage:  sum  0.00  ,  avg  0.00  ,  max  0.00  ,  min  0.00  
                                -  OpenTime:  avg  130.883us,  max  143.370us,  min  120.96us
                                -  ProjectionTime:  avg  420.824us,  max  636.882us,  min  763ns
                                -  RowsProduced:  sum  0,  avg  0,  max  0,  min  0
                                -  WaitForDependency[AGGREGATION_OPERATOR_DEPENDENCY]Time:  avg  72.547ms,  max  79.260ms,  min  65.118ms

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@jacktengg
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39893 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c44875a339a300b610ffde247bbc92d02047f38b, data reload: false

------ Round 1 ----------------------------------
q1	17652	4347	4377	4347
q2	2009	188	187	187
q3	10464	1173	1163	1163
q4	10186	738	873	738
q5	7518	2686	2714	2686
q6	219	137	136	136
q7	971	586	583	583
q8	9211	2064	2077	2064
q9	6870	6565	6537	6537
q10	8860	3752	3763	3752
q11	492	224	235	224
q12	460	222	220	220
q13	18932	3007	2947	2947
q14	283	228	226	226
q15	516	475	483	475
q16	491	386	392	386
q17	952	676	770	676
q18	8001	7437	7476	7437
q19	6474	1418	1388	1388
q20	647	322	322	322
q21	4862	3229	3119	3119
q22	340	280	282	280
Total cold run time: 116410 ms
Total hot run time: 39893 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4356	4251	4251	4251
q2	368	266	266	266
q3	3001	2794	2886	2794
q4	1979	1720	1693	1693
q5	5633	5562	5478	5478
q6	223	130	129	129
q7	2163	1829	1876	1829
q8	3292	3427	3454	3427
q9	8844	8766	8930	8766
q10	4050	3965	3761	3761
q11	590	508	492	492
q12	813	607	638	607
q13	17083	3164	3235	3164
q14	323	294	274	274
q15	545	500	498	498
q16	492	435	437	435
q17	1832	1537	1519	1519
q18	8074	8039	7775	7775
q19	1803	1588	1673	1588
q20	2902	1903	1886	1886
q21	10382	4842	4895	4842
q22	583	494	539	494
Total cold run time: 79331 ms
Total hot run time: 55968 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173913 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c44875a339a300b610ffde247bbc92d02047f38b, data reload: false

query1	921	375	361	361
query2	6453	1894	1775	1775
query3	6655	206	220	206
query4	27824	17391	17231	17231
query5	3649	479	489	479
query6	263	171	180	171
query7	4583	292	287	287
query8	240	187	188	187
query9	8536	2445	2415	2415
query10	454	286	280	280
query11	10909	9987	10073	9987
query12	116	95	82	82
query13	1646	380	388	380
query14	10267	7815	7702	7702
query15	219	167	167	167
query16	7621	476	493	476
query17	1150	562	536	536
query18	1742	286	294	286
query19	201	159	152	152
query20	90	83	83	83
query21	205	130	126	126
query22	4298	4099	3971	3971
query23	34038	33893	33727	33727
query24	8694	2941	2906	2906
query25	645	479	388	388
query26	715	150	151	150
query27	2239	288	290	288
query28	5748	2104	2078	2078
query29	905	639	629	629
query30	256	151	153	151
query31	961	752	774	752
query32	95	54	58	54
query33	601	349	330	330
query34	923	523	525	523
query35	875	760	777	760
query36	1112	961	957	957
query37	142	82	84	82
query38	2923	2857	2732	2732
query39	856	817	811	811
query40	194	115	125	115
query41	45	43	47	43
query42	114	102	99	99
query43	493	470	468	468
query44	1088	733	740	733
query45	196	162	161	161
query46	1092	722	761	722
query47	1861	1753	1758	1753
query48	363	293	290	290
query49	830	399	420	399
query50	785	400	401	400
query51	6813	6698	6666	6666
query52	106	90	90	90
query53	363	303	293	293
query54	636	447	449	447
query55	76	76	73	73
query56	291	258	283	258
query57	1101	1079	1038	1038
query58	252	243	253	243
query59	2883	2591	2581	2581
query60	318	271	305	271
query61	97	94	97	94
query62	793	648	650	648
query63	319	290	294	290
query64	9143	2230	1670	1670
query65	3125	3116	3146	3116
query66	748	332	329	329
query67	15564	15100	14980	14980
query68	4566	556	557	556
query69	694	416	350	350
query70	1172	1141	1114	1114
query71	427	283	286	283
query72	8878	5587	5696	5587
query73	757	327	330	327
query74	6254	5639	5572	5572
query75	3890	2724	2756	2724
query76	2917	1009	1019	1009
query77	691	306	320	306
query78	9769	8939	11295	8939
query79	2243	522	525	522
query80	2467	473	476	473
query81	586	218	215	215
query82	329	133	132	132
query83	316	171	168	168
query84	279	89	86	86
query85	1357	317	302	302
query86	447	332	313	313
query87	3292	3098	3063	3063
query88	3235	2383	2459	2383
query89	470	393	383	383
query90	1896	193	200	193
query91	131	99	101	99
query92	59	48	48	48
query93	1008	514	513	513
query94	1301	290	288	288
query95	399	319	316	316
query96	583	271	278	271
query97	3272	3039	3056	3039
query98	228	202	194	194
query99	1555	1217	1267	1217
Total cold run time: 274927 ms
Total hot run time: 173913 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c44875a339a300b610ffde247bbc92d02047f38b, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.06
query4	1.74	0.09	0.09
query5	0.50	0.50	0.48
query6	1.13	0.72	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.47
query10	0.52	0.54	0.53
query11	0.14	0.10	0.12
query12	0.14	0.12	0.12
query13	0.59	0.59	0.58
query14	0.75	0.78	0.80
query15	0.84	0.81	0.82
query16	0.34	0.36	0.34
query17	0.94	0.94	0.93
query18	0.23	0.22	0.21
query19	1.79	1.78	1.70
query20	0.02	0.01	0.01
query21	15.40	0.79	0.65
query22	4.44	6.87	1.75
query23	18.24	1.33	1.22
query24	2.09	0.22	0.21
query25	0.15	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.23	0.22
query28	13.37	1.01	1.00
query29	12.69	3.31	3.26
query30	0.25	0.08	0.06
query31	2.87	0.39	0.40
query32	3.24	0.46	0.47
query33	2.87	2.92	2.89
query34	16.94	4.32	4.34
query35	4.42	4.35	4.43
query36	0.64	0.46	0.48
query37	0.19	0.17	0.17
query38	0.16	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.05
Total cold run time: 109.71 s
Total hot run time: 30.19 s

@jacktengg jacktengg changed the title [fix](agg) fix RowsProduced is not set [fix](agg) fix RowsProduced counter is not set Jul 24, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

wangbo
wangbo approved these changes Jul 24, 2024
View reviewed changes
Copy link
Contributor

@wangbo wangbo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@HappenLee HappenLee merged commit ed4abc7 into apache:master Jul 24, 2024
31 of 34 checks passed
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@jacktengg @dataroaring
[fix](agg) fix RowsProduced counter is not set (#38271)
3c6a7ec 
## Proposed changes
`RowsProduced` and `BlocksProduced` counter of agg source is not right:

```
Pipeline  :  0(instance_num=4):
                      RESULT_SINK_OPERATOR  (id=0):
                            -  PlanInfo
                                  -  TABLE:  rqg_1333358549.table_10_undef_partitions2_keys3_properties4_distributed_by52(table_10_undef_partitions2_keys3_properties4_distributed_by52),  PREAGGREGATION:  ON
                                  -  partitions=1/1  (table_10_undef_partitions2_keys3_properties4_distributed_by52)
                                  -  tablets=10/10,  tabletList=39981436181982,39981436181984,39981436181986  ...
                                  -  cardinality=10,  avgRowSize=0.0,  numNodes=1
                                  -  pushAggOp=NONE
                            -  BlocksProduced:  sum  4,  avg  1,  max  1,  min  1
                            -  CloseTime:  avg  28.396us,  max  43.17us,  min  19.647us
                            -  ExecTime:  avg  514.688us,  max  694.677us,  min  353.811us
                            -  InitTime:  avg  52.136us,  max  55.309us,  min  46.966us
                            -  InputRows:  sum  0,  avg  0,  max  0,  min  0
                            -  MemoryUsage:  sum  ,  avg  ,  max  ,  min  
                                -  PeakMemoryUsage:  sum  0.00  ,  avg  0.00  ,  max  0.00  ,  min  0.00  
                            -  OpenTime:  avg  212.328us,  max  249.375us,  min  170.678us
                            -  RowsProduced:  sum  8,  avg  2,  max  3,  min  0
                            -  WaitForDependencyTime:  avg  0ns,  max  0ns,  min  0ns
                            -  WaitForDependency[RESULT_SINK_OPERATOR_DEPENDENCY]Time:  avg  0ns,  max  0ns,  min  0ns
                          AGGREGATION_OPERATOR  (id=10  ,  nereids_id=598):
                                -  PlanInfo
                                      -  output:  count(pk)[#31]
                                      -  group  by:  col_varchar_10__undef_signed
                                      -  sortByGroupKey:false
                                      -  cardinality=8
                                      -  projections:  field1,  col_varchar_10__undef_signed
                                      -  project  output  tuple  id:  11
                                -  BlocksProduced:  sum  0,  avg  0,  max  0,  min  0
                                -  CloseTime:  avg  5.617us,  max  6.543us,  min  5.247us
                                -  ExecTime:  avg  1.172ms,  max  1.609ms,  min  289.815us
                                -  InitTime:  avg  0ns,  max  0ns,  min  0ns
                                -  MemoryUsage:  sum  ,  avg  ,  max  ,  min  
                                    -  PeakMemoryUsage:  sum  0.00  ,  avg  0.00  ,  max  0.00  ,  min  0.00  
                                -  OpenTime:  avg  130.883us,  max  143.370us,  min  120.96us
                                -  ProjectionTime:  avg  420.824us,  max  636.882us,  min  763ns
                                -  RowsProduced:  sum  0,  avg  0,  max  0,  min  0
                                -  WaitForDependency[AGGREGATION_OPERATOR_DEPENDENCY]Time:  avg  72.547ms,  max  79.260ms,  min  65.118ms
```
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@wangbo wangbo wangbo approved these changes

@yiguolei yiguolei yiguolei approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@jacktengg @doris-robot @yiguolei @wangbo @Gabriel39 @HappenLee @dataroaring