Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Improvement](scan) Refine ignoring data distribution without join #37282

Merged
merged 4 commits into from
Jul 5, 2024
Merged

[Improvement](scan) Refine ignoring data distribution without join #37282

merged 4 commits into from
Jul 5, 2024

Conversation

Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Gabriel39
Copy link
Contributor Author

run buildall

1 similar comment
@Gabriel39
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39819 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b9c85d233cdb6207b77eb3dd6f156f1348bf16db, data reload: false

------ Round 1 ----------------------------------
q1	18217	5048	4271	4271
q2	2027	191	196	191
q3	10544	1142	1029	1029
q4	10272	830	801	801
q5	7494	2687	2666	2666
q6	220	136	136	136
q7	970	594	601	594
q8	9222	2102	2070	2070
q9	8838	6478	6458	6458
q10	9009	3737	3707	3707
q11	470	235	234	234
q12	497	239	236	236
q13	18685	2981	3019	2981
q14	258	230	223	223
q15	540	473	485	473
q16	523	380	384	380
q17	979	744	704	704
q18	8075	7476	7401	7401
q19	2053	1458	1455	1455
q20	654	319	325	319
q21	4965	3143	3913	3143
q22	400	347	347	347
Total cold run time: 114912 ms
Total hot run time: 39819 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4372	4264	4237	4237
q2	362	269	272	269
q3	2927	2756	2921	2756
q4	2001	1737	1705	1705
q5	5577	5574	5565	5565
q6	225	131	135	131
q7	2197	1822	1903	1822
q8	3289	3416	3410	3410
q9	8710	8686	8779	8686
q10	4091	3842	3766	3766
q11	580	493	487	487
q12	784	597	669	597
q13	15991	3173	3158	3158
q14	324	276	286	276
q15	534	495	485	485
q16	500	436	445	436
q17	1815	1502	1484	1484
q18	8032	7943	7885	7885
q19	3490	1633	1669	1633
q20	2075	1872	1908	1872
q21	5222	4947	4972	4947
q22	625	552	583	552
Total cold run time: 73723 ms
Total hot run time: 56159 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173534 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b9c85d233cdb6207b77eb3dd6f156f1348bf16db, data reload: false

query1	923	374	365	365
query2	6462	2477	2358	2358
query3	6639	203	215	203
query4	28149	17550	17165	17165
query5	3644	477	491	477
query6	270	163	156	156
query7	4598	293	292	292
query8	335	299	303	299
query9	8582	2431	2403	2403
query10	570	304	298	298
query11	10787	9872	10067	9872
query12	118	86	87	86
query13	1656	377	377	377
query14	10217	7523	7423	7423
query15	256	176	184	176
query16	7776	316	314	314
query17	1807	557	527	527
query18	1372	286	288	286
query19	192	153	173	153
query20	91	81	84	81
query21	206	140	124	124
query22	4222	4124	4001	4001
query23	34076	33877	33609	33609
query24	10832	2904	2876	2876
query25	613	413	417	413
query26	1040	161	156	156
query27	2549	321	331	321
query28	7118	2136	2124	2124
query29	909	641	678	641
query30	255	154	159	154
query31	966	759	777	759
query32	102	57	59	57
query33	769	328	319	319
query34	922	492	497	492
query35	745	664	647	647
query36	1168	1008	991	991
query37	152	82	86	82
query38	2914	2840	2798	2798
query39	882	825	827	825
query40	213	132	124	124
query41	55	54	141	54
query42	117	97	98	97
query43	583	558	561	558
query44	1197	723	733	723
query45	195	161	166	161
query46	1074	695	755	695
query47	1836	1773	1761	1761
query48	370	298	301	298
query49	826	402	423	402
query50	780	383	379	379
query51	6868	6851	6772	6772
query52	109	90	93	90
query53	356	289	285	285
query54	886	452	435	435
query55	74	72	70	70
query56	300	264	272	264
query57	1120	1043	1032	1032
query58	251	234	260	234
query59	3758	3072	3182	3072
query60	291	268	268	268
query61	97	93	92	92
query62	575	428	445	428
query63	308	294	295	294
query64	8811	2238	1741	1741
query65	3180	3089	3069	3069
query66	704	356	331	331
query67	15616	14940	14947	14940
query68	7746	529	529	529
query69	758	506	364	364
query70	1132	1144	1140	1140
query71	511	277	274	274
query72	8259	5456	5369	5369
query73	817	325	320	320
query74	5957	5478	5525	5478
query75	4961	2663	2692	2663
query76	5007	1033	1025	1025
query77	804	312	306	306
query78	9639	9179	8937	8937
query79	8419	537	527	527
query80	1134	479	478	478
query81	574	214	217	214
query82	739	110	103	103
query83	349	168	167	167
query84	280	85	89	85
query85	1455	369	310	310
query86	423	329	337	329
query87	3247	3137	3079	3079
query88	4175	2374	2372	2372
query89	505	372	378	372
query90	2121	192	189	189
query91	136	113	103	103
query92	63	49	50	49
query93	6452	517	510	510
query94	1380	213	209	209
query95	415	320	319	319
query96	604	268	268	268
query97	3201	3052	3034	3034
query98	230	198	202	198
query99	1158	846	845	845
Total cold run time: 298085 ms
Total hot run time: 173534 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b9c85d233cdb6207b77eb3dd6f156f1348bf16db, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.23	0.04	0.05
query4	1.68	0.07	0.08
query5	0.50	0.49	0.50
query6	1.13	0.74	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.55	0.49	0.48
query10	0.55	0.56	0.54
query11	0.16	0.11	0.11
query12	0.14	0.12	0.13
query13	0.59	0.59	0.59
query14	0.79	0.76	0.79
query15	0.85	0.81	0.82
query16	0.35	0.35	0.35
query17	0.99	1.01	0.96
query18	0.23	0.24	0.24
query19	1.74	1.73	1.70
query20	0.01	0.01	0.01
query21	15.38	0.77	0.66
query22	3.65	7.39	1.89
query23	18.28	1.39	1.23
query24	2.05	0.22	0.23
query25	0.16	0.08	0.08
query26	0.26	0.17	0.17
query27	0.07	0.08	0.07
query28	13.22	1.02	1.00
query29	12.62	3.33	3.34
query30	0.25	0.05	0.06
query31	2.88	0.41	0.40
query32	3.25	0.47	0.47
query33	2.91	2.95	2.86
query34	17.03	4.38	4.44
query35	4.49	4.54	4.59
query36	0.66	0.46	0.48
query37	0.18	0.15	0.16
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.18	0.14	0.15
query41	0.09	0.04	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 108.58 s
Total hot run time: 30.51 s

@Gabriel39
Copy link
Contributor Author

run buildall

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

zhangstar333
zhangstar333 previously approved these changes Jul 4, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@Gabriel39
Copy link
Contributor Author

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 40022 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c7534fc09d42bbfeea2a0b609ebcf9c1670753f, data reload: false

------ Round 1 ----------------------------------
q1	17608	4411	4310	4310
q2	2017	194	189	189
q3	10519	1245	1118	1118
q4	10193	778	684	684
q5	7495	2691	2663	2663
q6	228	140	140	140
q7	976	647	631	631
q8	9512	2095	2043	2043
q9	8899	6462	6454	6454
q10	8944	3746	3713	3713
q11	448	238	241	238
q12	554	233	233	233
q13	17770	2953	2961	2953
q14	263	224	232	224
q15	524	492	492	492
q16	503	379	377	377
q17	951	641	664	641
q18	8080	7537	7475	7475
q19	5504	1484	1564	1484
q20	686	338	335	335
q21	5004	3287	3809	3287
q22	391	338	348	338
Total cold run time: 117069 ms
Total hot run time: 40022 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4353	4229	4256	4229
q2	370	262	270	262
q3	2928	2761	2697	2697
q4	1883	1556	1637	1556
q5	5238	5241	5241	5241
q6	225	128	131	128
q7	2120	1708	1754	1708
q8	3207	3312	3280	3280
q9	8230	8300	8274	8274
q10	3805	3626	3662	3626
q11	594	487	486	486
q12	746	619	616	616
q13	16673	2970	2958	2958
q14	288	264	248	248
q15	518	487	471	471
q16	477	410	422	410
q17	1774	1499	1463	1463
q18	7686	7434	7425	7425
q19	3926	1446	1550	1446
q20	1956	1813	1803	1803
q21	4947	4586	4681	4586
q22	592	551	543	543
Total cold run time: 72536 ms
Total hot run time: 53456 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172468 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c7534fc09d42bbfeea2a0b609ebcf9c1670753f, data reload: false

query1	908	373	363	363
query2	6464	2400	2265	2265
query3	6658	217	218	217
query4	28326	17542	17492	17492
query5	4191	504	475	475
query6	284	182	174	174
query7	4608	300	301	300
query8	326	295	314	295
query9	8686	2422	2392	2392
query10	607	291	309	291
query11	10772	9954	10195	9954
query12	140	88	82	82
query13	1645	395	378	378
query14	9707	6963	7546	6963
query15	236	184	184	184
query16	7781	310	296	296
query17	1817	555	515	515
query18	1905	273	271	271
query19	198	143	145	143
query20	91	80	78	78
query21	217	124	126	124
query22	4555	4238	4317	4238
query23	33766	33125	32968	32968
query24	11989	2867	2854	2854
query25	665	375	375	375
query26	1716	152	151	151
query27	2943	322	318	318
query28	7622	2063	2049	2049
query29	1037	626	615	615
query30	286	148	147	147
query31	969	739	750	739
query32	99	52	61	52
query33	772	326	305	305
query34	906	485	488	485
query35	723	641	613	613
query36	1070	933	942	933
query37	142	75	73	73
query38	2885	2754	2717	2717
query39	853	795	819	795
query40	276	128	124	124
query41	55	51	56	51
query42	116	99	100	99
query43	588	527	531	527
query44	1229	746	718	718
query45	189	163	164	163
query46	1082	707	714	707
query47	1894	1776	1794	1776
query48	372	301	298	298
query49	1156	410	407	407
query50	770	392	384	384
query51	6864	6820	6794	6794
query52	101	94	94	94
query53	358	304	299	299
query54	943	444	453	444
query55	78	72	75	72
query56	295	271	302	271
query57	1146	1047	1069	1047
query58	264	232	250	232
query59	3308	3182	3152	3152
query60	302	291	283	283
query61	98	96	93	93
query62	636	435	442	435
query63	314	301	285	285
query64	9980	2232	1794	1794
query65	3181	3107	3133	3107
query66	1402	338	347	338
query67	15344	14971	15070	14971
query68	4615	526	529	526
query69	563	430	331	331
query70	1119	1054	1156	1054
query71	419	279	277	277
query72	7513	5909	5252	5252
query73	768	331	322	322
query74	5906	5573	5506	5506
query75	3346	2695	2656	2656
query76	2792	949	978	949
query77	484	304	312	304
query78	9450	9040	9875	9040
query79	2545	504	508	504
query80	1877	468	511	468
query81	597	211	215	211
query82	763	107	106	106
query83	297	164	165	164
query84	264	85	83	83
query85	1239	317	291	291
query86	445	329	318	318
query87	3230	3088	3153	3088
query88	3975	2369	2378	2369
query89	471	373	381	373
query90	1838	192	182	182
query91	127	103	106	103
query92	66	50	51	50
query93	1876	501	507	501
query94	1212	207	207	207
query95	399	309	312	309
query96	592	271	274	271
query97	3162	3004	2978	2978
query98	219	207	193	193
query99	1166	838	860	838
Total cold run time: 284241 ms
Total hot run time: 172468 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c7534fc09d42bbfeea2a0b609ebcf9c1670753f, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.06	0.05
query4	1.66	0.07	0.08
query5	0.50	0.47	0.48
query6	1.15	0.72	0.73
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.48
query10	0.53	0.54	0.54
query11	0.15	0.11	0.12
query12	0.15	0.12	0.13
query13	0.58	0.58	0.59
query14	0.78	0.76	0.78
query15	0.85	0.81	0.82
query16	0.37	0.37	0.37
query17	1.01	1.05	0.99
query18	0.22	0.26	0.22
query19	1.88	1.75	1.70
query20	0.02	0.01	0.01
query21	15.42	0.75	0.67
query22	4.53	6.96	2.13
query23	18.25	1.37	1.27
query24	2.13	0.24	0.23
query25	0.15	0.09	0.08
query26	0.30	0.20	0.21
query27	0.47	0.23	0.24
query28	13.22	1.01	0.99
query29	12.61	3.43	3.37
query30	0.25	0.06	0.05
query31	2.87	0.39	0.41
query32	3.23	0.47	0.46
query33	2.90	2.92	2.90
query34	17.04	4.32	4.38
query35	4.46	4.43	4.43
query36	0.64	0.48	0.47
query37	0.18	0.15	0.15
query38	0.15	0.14	0.14
query39	0.05	0.03	0.03
query40	0.15	0.13	0.12
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.03	0.04
Total cold run time: 110.01 s
Total hot run time: 30.88 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 560e352 into apache:master Jul 5, 2024
28 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@Gabriel39 @dataroaring
[Improvement](scan) Refine ignoring data distribution without join (#…
baa4d90 
…37282)

Refine ignoring data distribution without join
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangstar333 zhangstar333 zhangstar333 approved these changes

@Mryange Mryange Mryange approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@Gabriel39 @doris-robot @Mryange @zhangstar333 @BiteTheDDDDt @dataroaring