Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[opt](nereids) recover adoptive bucket shuffle #36784

Merged
merged 2 commits into from
Jul 3, 2024
Merged

[opt](nereids) recover adoptive bucket shuffle #36784

merged 2 commits into from
Jul 3, 2024

Conversation

xzj7019
Copy link
Contributor

@xzj7019 xzj7019 commented Jun 25, 2024

Proposed changes

  1. Recover adoptive bucket shuffle and re-using the enable_bucket_shuffle_join to control whether to enable it, default remains true.
  2. Remove enable_bucket_shuffle_downgrade option.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@xzj7019 xzj7019 marked this pull request as draft June 25, 2024 03:01
@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from 917e3ba to 7cb2b14 Compare June 25, 2024 08:59
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 25, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40420 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7cb2b14b6a6c3919ccdc9ab51a1d178e1c01f658, data reload: false

------ Round 1 ----------------------------------
q1	17586	4344	4342	4342
q2	2007	189	185	185
q3	10451	1185	1176	1176
q4	10186	735	820	735
q5	7452	2635	2601	2601
q6	221	140	132	132
q7	942	600	605	600
q8	9231	2059	2056	2056
q9	9019	6528	6516	6516
q10	8937	3728	3687	3687
q11	449	243	239	239
q12	575	230	225	225
q13	17775	2961	2978	2961
q14	273	224	230	224
q15	528	473	478	473
q16	518	379	374	374
q17	958	639	635	635
q18	7968	7473	7318	7318
q19	4542	1447	1530	1447
q20	655	322	315	315
q21	5085	3825	3939	3825
q22	400	354	357	354
Total cold run time: 115758 ms
Total hot run time: 40420 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4405	4245	4349	4245
q2	362	254	263	254
q3	2972	2847	2923	2847
q4	2006	1679	1652	1652
q5	5664	5532	5481	5481
q6	220	127	129	127
q7	2209	1911	1903	1903
q8	3290	3392	3488	3392
q9	8690	8760	8799	8760
q10	4100	3801	3816	3801
q11	576	528	512	512
q12	861	634	637	634
q13	16062	3193	3232	3193
q14	301	293	278	278
q15	513	484	492	484
q16	497	438	451	438
q17	1825	1523	1519	1519
q18	8174	7858	7916	7858
q19	4268	1554	1709	1554
q20	3153	1911	1851	1851
q21	5207	4958	4879	4879
q22	656	570	565	565
Total cold run time: 76011 ms
Total hot run time: 56227 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174668 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7cb2b14b6a6c3919ccdc9ab51a1d178e1c01f658, data reload: false

query1	907	396	378	378
query2	6633	2363	2357	2357
query3	6628	218	219	218
query4	19746	17436	17386	17386
query5	3616	503	471	471
query6	245	164	166	164
query7	4586	310	291	291
query8	341	297	292	292
query9	8668	2395	2373	2373
query10	567	287	287	287
query11	10653	10004	10182	10004
query12	110	88	92	88
query13	1638	365	387	365
query14	9429	6959	6926	6926
query15	236	194	202	194
query16	7416	288	279	279
query17	1423	552	546	546
query18	1848	285	281	281
query19	202	165	161	161
query20	91	84	85	84
query21	214	133	126	126
query22	4243	4174	4064	4064
query23	33956	33966	33702	33702
query24	11486	2845	2859	2845
query25	648	420	405	405
query26	1236	171	168	168
query27	3085	333	335	333
query28	7868	2162	2143	2143
query29	917	684	653	653
query30	251	162	211	162
query31	934	760	796	760
query32	94	52	57	52
query33	750	300	309	300
query34	979	501	484	484
query35	765	635	659	635
query36	1104	981	995	981
query37	172	79	75	75
query38	2950	2892	2828	2828
query39	870	803	782	782
query40	217	129	130	129
query41	56	55	57	55
query42	121	99	101	99
query43	593	555	543	543
query44	1239	737	726	726
query45	189	163	166	163
query46	1079	732	712	712
query47	1852	1755	1764	1755
query48	374	302	304	302
query49	859	420	440	420
query50	785	394	386	386
query51	6891	6782	6684	6684
query52	111	91	96	91
query53	364	298	293	293
query54	884	446	445	445
query55	76	76	74	74
query56	290	274	285	274
query57	1141	1067	1056	1056
query58	271	262	259	259
query59	3693	3241	3114	3114
query60	291	293	287	287
query61	97	96	99	96
query62	611	426	450	426
query63	326	308	298	298
query64	8918	2267	1779	1779
query65	3206	3127	3187	3127
query66	864	345	325	325
query67	15475	15141	15018	15018
query68	4521	553	542	542
query69	602	410	421	410
query70	1186	1139	1085	1085
query71	435	286	282	282
query72	8935	5554	5648	5554
query73	750	329	327	327
query74	5847	5475	5457	5457
query75	3562	2710	2682	2682
query76	2486	981	942	942
query77	496	318	312	312
query78	10327	9972	9804	9804
query79	3065	515	522	515
query80	1910	466	463	463
query81	571	224	222	222
query82	1279	110	107	107
query83	338	180	181	180
query84	271	89	84	84
query85	1501	355	282	282
query86	459	321	299	299
query87	3297	3112	3116	3112
query88	4304	2371	2365	2365
query89	470	372	388	372
query90	1807	185	181	181
query91	132	101	102	101
query92	62	48	52	48
query93	3452	525	523	523
query94	1227	227	185	185
query95	396	313	309	309
query96	607	266	263	263
query97	3283	3041	3024	3024
query98	225	198	195	195
query99	1172	864	862	862
Total cold run time: 277353 ms
Total hot run time: 174668 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.26 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7cb2b14b6a6c3919ccdc9ab51a1d178e1c01f658, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.09
query5	0.49	0.48	0.50
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.51
query10	0.54	0.53	0.55
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.59	0.58	0.60
query14	0.77	0.76	0.79
query15	0.83	0.80	0.81
query16	0.37	0.35	0.35
query17	0.97	0.95	1.00
query18	0.23	0.26	0.23
query19	1.87	1.69	1.64
query20	0.01	0.01	0.01
query21	15.46	0.66	0.64
query22	4.39	7.82	1.82
query23	18.43	1.41	1.21
query24	2.14	0.22	0.22
query25	0.16	0.08	0.08
query26	0.27	0.18	0.18
query27	0.08	0.08	0.07
query28	13.21	1.02	1.01
query29	12.62	3.29	3.27
query30	0.25	0.08	0.06
query31	2.85	0.40	0.40
query32	3.24	0.48	0.47
query33	2.84	2.94	2.90
query34	17.26	4.39	4.45
query35	4.52	4.46	4.52
query36	0.66	0.47	0.49
query37	0.17	0.15	0.15
query38	0.15	0.14	0.15
query39	0.05	0.04	0.03
query40	0.17	0.15	0.14
query41	0.09	0.04	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.85 s
Total hot run time: 30.26 s

@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from c814ff0 to f1b0fe6 Compare June 26, 2024 08:29
@xzj7019 xzj7019 marked this pull request as ready for review June 26, 2024 08:31
@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from f1b0fe6 to c1cd82e Compare June 26, 2024 08:31
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 26, 2024

run buildall

morrySnow
morrySnow reviewed Jun 26, 2024
View reviewed changes
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java Outdated
// - base table and tablets' number is small enough (< paraInstanceNum)
// otherSide:
// - ShuffleType.EXECUTION_BUCKETED
boolean isBucketShuffleDownGrade = ConnectContext.get().getSessionVariable().isEnableBucketShuffleJoin();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when enableBucketShuffleJoin = false, we should never do bucket shuffle at all. so when isEnableBucketShuffleJoin = false, function isBucketShuffleDownGrade should always return true?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

forget to change the name, isBucketShuffleDownGrade should be isEnableBucketShuffle.

fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildrenPropertiesRegulator.java Show resolved Hide resolved
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 26, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39526 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 85c32a531dfd5fe927ca744658de9a1fadc653e2, data reload: false

------ Round 1 ----------------------------------
q1	17607	4366	4291	4291
q2	2007	200	189	189
q3	10468	1110	1087	1087
q4	10186	740	814	740
q5	7494	2651	2603	2603
q6	219	136	134	134
q7	936	593	607	593
q8	9242	2048	2063	2048
q9	8762	6500	6436	6436
q10	9067	3735	3704	3704
q11	508	237	236	236
q12	475	232	223	223
q13	17771	2998	2973	2973
q14	255	216	238	216
q15	523	469	481	469
q16	513	385	371	371
q17	951	644	650	644
q18	7961	7368	7379	7368
q19	5851	1437	1428	1428
q20	658	319	319	319
q21	4918	3114	3916	3114
q22	392	340	342	340
Total cold run time: 116764 ms
Total hot run time: 39526 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4499	4300	4234	4234
q2	377	252	261	252
q3	2962	2882	2844	2844
q4	1971	1716	1703	1703
q5	5630	5500	5489	5489
q6	210	126	131	126
q7	2216	1898	1862	1862
q8	3265	3383	3434	3383
q9	8688	8653	8848	8653
q10	4141	3891	3704	3704
q11	592	499	497	497
q12	819	656	656	656
q13	16242	3178	3141	3141
q14	307	279	284	279
q15	521	479	496	479
q16	473	435	437	435
q17	1871	1511	1493	1493
q18	8101	7787	7837	7787
q19	1783	1573	1675	1573
q20	3046	1888	1895	1888
q21	5058	4915	4877	4877
q22	630	558	565	558
Total cold run time: 73402 ms
Total hot run time: 55913 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173632 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 85c32a531dfd5fe927ca744658de9a1fadc653e2, data reload: false

query1	909	399	359	359
query2	6444	2389	2257	2257
query3	6616	207	207	207
query4	18821	17010	17379	17010
query5	3655	463	467	463
query6	251	163	180	163
query7	4586	301	304	301
query8	313	292	291	291
query9	8524	2465	2441	2441
query10	570	314	315	314
query11	10489	10126	10040	10040
query12	109	83	82	82
query13	1628	368	372	368
query14	10255	7667	7035	7035
query15	239	190	186	186
query16	7808	272	289	272
query17	1871	543	505	505
query18	1954	273	267	267
query19	185	151	157	151
query20	91	81	81	81
query21	202	130	120	120
query22	4416	4067	4056	4056
query23	33907	33730	33669	33669
query24	11067	2820	2913	2820
query25	603	381	384	381
query26	717	166	153	153
query27	2307	323	329	323
query28	6244	2165	2167	2165
query29	896	665	630	630
query30	245	159	165	159
query31	1002	734	750	734
query32	100	53	55	53
query33	774	276	310	276
query34	1041	502	475	475
query35	762	619	647	619
query36	1147	972	969	969
query37	135	81	72	72
query38	2934	2873	2816	2816
query39	877	839	850	839
query40	219	135	132	132
query41	60	56	53	53
query42	128	98	98	98
query43	588	541	534	534
query44	1192	752	741	741
query45	205	166	166	166
query46	1085	735	738	735
query47	1843	1792	1746	1746
query48	384	316	337	316
query49	828	437	431	431
query50	779	394	402	394
query51	6930	6801	6716	6716
query52	110	93	95	93
query53	359	292	295	292
query54	881	466	442	442
query55	74	73	73	73
query56	280	265	259	259
query57	1151	1069	1054	1054
query58	253	242	254	242
query59	3413	3198	3096	3096
query60	293	273	284	273
query61	95	89	90	89
query62	603	440	456	440
query63	322	297	298	297
query64	8566	2293	1777	1777
query65	3178	3078	3109	3078
query66	736	322	331	322
query67	15389	14959	14766	14766
query68	4831	523	542	523
query69	580	494	403	403
query70	1211	1106	1176	1106
query71	440	288	279	279
query72	7977	5368	5370	5368
query73	769	322	323	322
query74	5966	5577	5488	5488
query75	3472	2633	2649	2633
query76	2996	951	893	893
query77	613	367	300	300
query78	10541	9792	9798	9792
query79	2418	522	514	514
query80	2415	478	469	469
query81	588	221	218	218
query82	1442	108	106	106
query83	302	171	175	171
query84	258	88	88	88
query85	1336	290	286	286
query86	469	338	295	295
query87	3233	3058	3077	3058
query88	4065	2464	2452	2452
query89	482	390	405	390
query90	1712	193	188	188
query91	135	101	103	101
query92	58	50	49	49
query93	2303	511	504	504
query94	1061	188	188	188
query95	416	322	379	322
query96	601	275	268	268
query97	3251	3046	3042	3042
query98	224	203	196	196
query99	1189	858	843	843
Total cold run time: 271992 ms
Total hot run time: 173632 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 85c32a531dfd5fe927ca744658de9a1fadc653e2, data reload: false

query1	0.05	0.04	0.04
query2	0.07	0.04	0.04
query3	0.22	0.05	0.04
query4	1.69	0.06	0.08
query5	0.50	0.48	0.47
query6	1.13	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.03
query9	0.55	0.49	0.49
query10	0.54	0.54	0.54
query11	0.16	0.11	0.11
query12	0.15	0.12	0.12
query13	0.59	0.58	0.59
query14	0.75	0.79	0.80
query15	0.84	0.82	0.81
query16	0.37	0.37	0.37
query17	1.03	1.03	1.03
query18	0.22	0.23	0.26
query19	1.80	1.69	1.84
query20	0.02	0.01	0.01
query21	15.42	0.66	0.65
query22	4.45	6.79	2.13
query23	18.21	1.38	1.31
query24	2.12	0.22	0.23
query25	0.16	0.08	0.08
query26	0.28	0.18	0.17
query27	0.08	0.09	0.09
query28	13.24	1.04	0.99
query29	12.65	3.29	3.29
query30	0.27	0.06	0.06
query31	2.86	0.38	0.38
query32	3.29	0.47	0.47
query33	2.82	2.89	2.86
query34	17.17	4.38	4.37
query35	4.46	4.48	4.54
query36	0.66	0.46	0.50
query37	0.18	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.16	0.14	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.05	0.03
Total cold run time: 109.64 s
Total hot run time: 30.81 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 26, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40326 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 310a3a0f172174ef3b9fb83864f383dae076a771, data reload: false

------ Round 1 ----------------------------------
q1	17604	4393	4277	4277
q2	2017	186	180	180
q3	10472	1134	1059	1059
q4	10180	781	813	781
q5	7447	2645	2645	2645
q6	217	133	134	133
q7	940	600	605	600
q8	9234	2059	2072	2059
q9	8727	6438	6427	6427
q10	8946	3671	3710	3671
q11	440	245	241	241
q12	426	235	227	227
q13	18955	2982	2980	2980
q14	259	221	219	219
q15	518	479	484	479
q16	498	386	376	376
q17	957	710	700	700
q18	7891	7366	7400	7366
q19	6305	1436	1354	1354
q20	641	338	319	319
q21	4977	3890	3960	3890
q22	395	343	350	343
Total cold run time: 118046 ms
Total hot run time: 40326 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4501	4252	4285	4252
q2	374	253	264	253
q3	2967	2892	2962	2892
q4	1980	1650	1674	1650
q5	5528	5473	5471	5471
q6	220	140	130	130
q7	2264	1883	1862	1862
q8	3283	3418	3432	3418
q9	8708	8707	8720	8707
q10	4155	3713	3759	3713
q11	614	509	495	495
q12	850	609	647	609
q13	16116	3239	3209	3209
q14	298	279	271	271
q15	525	485	503	485
q16	500	455	433	433
q17	1800	1586	1474	1474
q18	8020	8041	7838	7838
q19	1792	1631	1652	1631
q20	3184	1907	1871	1871
q21	5109	5071	4784	4784
q22	669	565	574	565
Total cold run time: 73457 ms
Total hot run time: 56013 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173538 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 310a3a0f172174ef3b9fb83864f383dae076a771, data reload: false

query1	910	384	373	373
query2	6407	2382	2303	2303
query3	6633	206	199	199
query4	18846	17342	17240	17240
query5	3607	455	476	455
query6	237	160	158	158
query7	4596	296	284	284
query8	317	290	278	278
query9	8689	2460	2448	2448
query10	553	296	276	276
query11	10607	10184	10078	10078
query12	110	87	82	82
query13	1629	378	369	369
query14	8600	6957	6244	6244
query15	241	195	181	181
query16	7125	277	261	261
query17	1013	533	515	515
query18	1909	265	264	264
query19	191	152	150	150
query20	84	81	81	81
query21	201	126	119	119
query22	4265	3985	4150	3985
query23	33855	33654	33816	33654
query24	11305	2887	2829	2829
query25	630	393	364	364
query26	812	159	159	159
query27	2386	367	331	331
query28	6384	2140	2146	2140
query29	914	609	627	609
query30	261	153	160	153
query31	1002	759	759	759
query32	93	60	54	54
query33	777	306	298	298
query34	1070	471	471	471
query35	768	622	622	622
query36	1180	976	983	976
query37	145	79	68	68
query38	2935	2821	2881	2821
query39	915	834	849	834
query40	224	133	127	127
query41	54	56	55	55
query42	139	104	100	100
query43	613	568	562	562
query44	1232	731	726	726
query45	191	166	163	163
query46	1083	694	718	694
query47	1896	1812	1753	1753
query48	367	303	297	297
query49	852	402	409	402
query50	771	393	396	393
query51	6930	6781	6782	6781
query52	107	88	96	88
query53	353	283	285	283
query54	899	440	436	436
query55	74	73	73	73
query56	282	272	272	272
query57	1145	1040	1021	1021
query58	272	243	244	243
query59	3348	3186	3169	3169
query60	325	264	277	264
query61	95	93	91	91
query62	621	463	447	447
query63	313	290	291	290
query64	8705	2235	1789	1789
query65	3160	3128	3079	3079
query66	832	342	332	332
query67	15276	15108	14954	14954
query68	5146	525	533	525
query69	606	496	429	429
query70	1154	1128	1168	1128
query71	441	279	286	279
query72	8095	5707	5640	5640
query73	778	327	327	327
query74	5891	5574	5597	5574
query75	3479	2647	2669	2647
query76	3053	884	949	884
query77	592	306	296	296
query78	10473	9805	9670	9670
query79	3498	516	514	514
query80	2109	492	483	483
query81	605	221	220	220
query82	1389	108	108	108
query83	322	187	177	177
query84	265	94	91	91
query85	1388	350	325	325
query86	470	307	308	307
query87	3330	3059	3057	3057
query88	4562	2611	2446	2446
query89	473	364	372	364
query90	1671	187	188	187
query91	130	101	99	99
query92	62	48	49	48
query93	3831	508	500	500
query94	1068	188	183	183
query95	402	313	318	313
query96	591	271	265	265
query97	3254	3051	3089	3051
query98	221	196	194	194
query99	1157	838	839	838
Total cold run time: 272867 ms
Total hot run time: 173538 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.67 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 310a3a0f172174ef3b9fb83864f383dae076a771, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.04
query3	0.22	0.05	0.05
query4	1.69	0.07	0.07
query5	0.49	0.48	0.50
query6	1.14	0.72	0.72
query7	0.01	0.01	0.02
query8	0.05	0.05	0.04
query9	0.52	0.50	0.49
query10	0.54	0.53	0.53
query11	0.15	0.11	0.11
query12	0.14	0.12	0.11
query13	0.59	0.58	0.59
query14	0.75	0.77	0.80
query15	0.83	0.81	0.80
query16	0.36	0.35	0.36
query17	0.95	0.98	0.98
query18	0.23	0.24	0.22
query19	1.88	1.67	1.70
query20	0.01	0.01	0.01
query21	15.43	0.65	0.66
query22	4.06	6.93	2.12
query23	18.30	1.29	1.34
query24	2.08	0.22	0.21
query25	0.14	0.09	0.09
query26	0.25	0.17	0.18
query27	0.08	0.08	0.08
query28	13.29	1.02	1.01
query29	12.63	3.26	3.26
query30	0.26	0.06	0.05
query31	2.88	0.37	0.38
query32	3.31	0.46	0.46
query33	2.82	2.94	2.88
query34	17.15	4.44	4.43
query35	4.44	4.48	4.53
query36	0.66	0.47	0.48
query37	0.17	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.03
query40	0.16	0.15	0.14
query41	0.10	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.16 s
Total hot run time: 30.67 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 27, 2024

run buildall

@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from 75daeb8 to 21b461a Compare June 27, 2024 01:34
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 27, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39733 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 21b461a604c72812b6409260e48caffe1fba3a25, data reload: false

------ Round 1 ----------------------------------
q1	17624	4385	4240	4240
q2	2023	194	200	194
q3	10460	1193	1107	1107
q4	10195	809	806	806
q5	7499	2664	2643	2643
q6	220	136	139	136
q7	950	595	603	595
q8	9249	2099	2077	2077
q9	9256	6489	6502	6489
q10	8926	3766	3670	3670
q11	454	243	229	229
q12	478	233	246	233
q13	18894	2948	2965	2948
q14	274	232	226	226
q15	501	490	496	490
q16	508	372	383	372
q17	966	616	665	616
q18	7886	7446	7359	7359
q19	5276	1478	1505	1478
q20	651	321	344	321
q21	4916	3168	3202	3168
q22	396	336	340	336
Total cold run time: 117602 ms
Total hot run time: 39733 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4394	4370	4251	4251
q2	365	264	261	261
q3	3002	2843	2903	2843
q4	2050	1716	1721	1716
q5	5639	5506	5475	5475
q6	225	128	127	127
q7	2238	1862	1853	1853
q8	3307	3403	3434	3403
q9	8626	8695	8813	8695
q10	4106	3827	3697	3697
q11	595	507	485	485
q12	865	644	646	644
q13	17043	3183	3154	3154
q14	289	266	281	266
q15	517	488	477	477
q16	501	432	449	432
q17	1820	1531	1531	1531
q18	8130	7982	7758	7758
q19	1765	1617	1507	1507
q20	3139	1887	1819	1819
q21	5250	4972	4794	4794
q22	982	573	537	537
Total cold run time: 74848 ms
Total hot run time: 55725 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174220 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 21b461a604c72812b6409260e48caffe1fba3a25, data reload: false

query1	906	376	376	376
query2	6458	2498	2322	2322
query3	6626	203	210	203
query4	18924	17469	17057	17057
query5	3611	471	453	453
query6	255	175	162	162
query7	4595	312	286	286
query8	310	299	281	281
query9	8948	2472	2455	2455
query10	576	317	285	285
query11	10538	9937	10161	9937
query12	114	87	83	83
query13	1646	380	374	374
query14	10230	7488	7673	7488
query15	230	189	194	189
query16	7833	291	276	276
query17	1529	567	550	550
query18	1967	286	283	283
query19	197	153	160	153
query20	95	83	87	83
query21	216	131	132	131
query22	4259	4035	3990	3990
query23	33857	33417	33808	33417
query24	11215	2867	2913	2867
query25	617	427	391	391
query26	717	165	161	161
query27	2329	328	330	328
query28	6084	2202	2186	2186
query29	879	683	663	663
query30	265	168	154	154
query31	993	769	766	766
query32	101	59	54	54
query33	729	308	311	308
query34	923	486	492	486
query35	779	661	641	641
query36	1118	982	983	982
query37	153	74	78	74
query38	2990	2820	2850	2820
query39	914	838	851	838
query40	216	127	142	127
query41	55	58	55	55
query42	117	106	113	106
query43	599	572	544	544
query44	1207	752	738	738
query45	196	178	166	166
query46	1091	767	725	725
query47	1850	1760	1794	1760
query48	461	310	306	306
query49	834	400	412	400
query50	790	396	389	389
query51	6877	6731	6745	6731
query52	105	94	92	92
query53	360	296	287	287
query54	895	456	436	436
query55	75	77	73	73
query56	275	256	260	256
query57	1129	1036	1036	1036
query58	255	253	228	228
query59	3528	3442	3084	3084
query60	304	287	286	286
query61	93	90	90	90
query62	589	437	444	437
query63	315	300	300	300
query64	8579	2297	1759	1759
query65	3130	3101	3103	3101
query66	747	327	328	327
query67	15719	15170	14836	14836
query68	8113	566	549	549
query69	694	510	440	440
query70	1203	1108	1163	1108
query71	551	272	276	272
query72	9034	5370	5567	5370
query73	859	331	325	325
query74	5840	5509	5551	5509
query75	5318	2697	2674	2674
query76	4983	948	953	948
query77	771	297	298	297
query78	10470	9911	9751	9751
query79	8067	517	522	517
query80	1030	460	465	460
query81	552	226	221	221
query82	697	109	104	104
query83	352	167	166	166
query84	279	82	89	82
query85	1286	291	276	276
query86	409	337	321	321
query87	3354	3140	3087	3087
query88	4840	2469	2476	2469
query89	511	385	379	379
query90	1874	194	188	188
query91	132	100	100	100
query92	61	49	50	49
query93	5715	508	507	507
query94	1081	187	191	187
query95	402	315	323	315
query96	614	266	267	266
query97	3204	3062	3042	3042
query98	220	200	196	196
query99	1278	817	834	817
Total cold run time: 288911 ms
Total hot run time: 174220 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 21b461a604c72812b6409260e48caffe1fba3a25, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.67	0.08	0.07
query5	0.51	0.47	0.48
query6	1.13	0.73	0.72
query7	0.03	0.01	0.02
query8	0.05	0.05	0.05
query9	0.55	0.50	0.50
query10	0.54	0.54	0.54
query11	0.15	0.12	0.12
query12	0.14	0.11	0.12
query13	0.60	0.59	0.61
query14	0.80	0.79	0.78
query15	0.86	0.82	0.81
query16	0.35	0.35	0.35
query17	0.98	0.99	0.97
query18	0.21	0.27	0.24
query19	1.91	1.73	1.70
query20	0.01	0.01	0.01
query21	15.42	0.73	0.66
query22	4.10	7.17	2.22
query23	18.30	1.35	1.28
query24	2.14	0.21	0.23
query25	0.16	0.08	0.10
query26	0.27	0.18	0.18
query27	0.09	0.08	0.08
query28	13.26	1.03	0.99
query29	12.61	3.38	3.35
query30	0.26	0.07	0.06
query31	2.86	0.39	0.38
query32	3.28	0.47	0.47
query33	2.83	2.91	2.94
query34	17.12	4.38	4.37
query35	4.49	4.56	4.52
query36	0.65	0.46	0.49
query37	0.18	0.16	0.16
query38	0.16	0.15	0.14
query39	0.04	0.04	0.04
query40	0.17	0.15	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.42 s
Total hot run time: 31 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 27, 2024

run buildall

@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from ae9d72a to 0adcecb Compare June 28, 2024 03:56
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jun 28, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40099 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0adcecbf4e5f48ace51732babc249754c442ef00, data reload: false

------ Round 1 ----------------------------------
q1	17904	6038	4336	4336
q2	2036	200	200	200
q3	10552	1183	1056	1056
q4	10329	756	848	756
q5	7483	2699	2669	2669
q6	223	143	138	138
q7	964	598	612	598
q8	9251	2130	2102	2102
q9	8980	6496	6460	6460
q10	9011	3778	3725	3725
q11	453	233	236	233
q12	403	237	234	234
q13	17918	2979	3020	2979
q14	273	226	225	225
q15	512	481	498	481
q16	502	374	380	374
q17	982	696	693	693
q18	8101	7422	7410	7410
q19	6518	1612	1483	1483
q20	656	323	333	323
q21	4854	3290	3802	3290
q22	401	334	343	334
Total cold run time: 118306 ms
Total hot run time: 40099 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4436	4271	4236	4236
q2	377	274	256	256
q3	3201	2921	2979	2921
q4	1975	1706	1728	1706
q5	5551	5480	5517	5480
q6	237	138	139	138
q7	2247	1833	1834	1833
q8	3313	3435	3438	3435
q9	8737	8780	8749	8749
q10	4161	3761	3816	3761
q11	582	525	501	501
q12	839	624	643	624
q13	17316	3187	3185	3185
q14	329	271	282	271
q15	558	476	494	476
q16	514	416	445	416
q17	1820	1507	1508	1507
q18	8320	8085	7989	7989
q19	1869	1488	1636	1488
q20	2079	1877	1859	1859
q21	8501	4801	4945	4801
q22	675	581	543	543
Total cold run time: 77637 ms
Total hot run time: 56175 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.71 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4303423b590758c2b10e7ac07992c802e9fd16fb, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.05	0.04
query3	0.22	0.04	0.04
query4	1.67	0.08	0.07
query5	0.50	0.48	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.48
query10	0.55	0.54	0.54
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.60	0.58	0.58
query14	0.76	0.78	0.78
query15	0.86	0.82	0.81
query16	0.37	0.37	0.35
query17	1.04	1.02	1.04
query18	0.23	0.24	0.24
query19	1.73	1.67	1.78
query20	0.01	0.01	0.02
query21	15.83	0.76	0.66
query22	4.17	7.17	2.02
query23	18.30	1.37	1.28
query24	2.11	0.24	0.24
query25	0.16	0.08	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.23	1.00	0.99
query29	12.66	3.36	3.33
query30	0.25	0.06	0.05
query31	2.87	0.38	0.40
query32	3.27	0.48	0.46
query33	2.83	2.90	2.90
query34	17.15	4.44	4.40
query35	4.51	4.45	4.50
query36	0.66	0.48	0.49
query37	0.19	0.16	0.16
query38	0.15	0.14	0.15
query39	0.04	0.03	0.03
query40	0.16	0.15	0.15
query41	0.09	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.04	0.04
Total cold run time: 109.79 s
Total hot run time: 30.71 s

morrySnow
morrySnow previously approved these changes Jul 3, 2024
View reviewed changes
@xzj7019 xzj7019 force-pushed the recover_adoptive_bucket_shuffle branch from 4303423 to 450f4c6 Compare July 3, 2024 07:14
@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40177 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 450f4c6ca901ea69501a0b840e7368dc50eddfbc, data reload: false

------ Round 1 ----------------------------------
q1	17833	4498	4417	4417
q2	2360	205	200	200
q3	10404	1198	1068	1068
q4	10190	743	829	743
q5	7526	2638	2637	2637
q6	227	139	138	138
q7	957	616	619	616
q8	9242	2080	2099	2080
q9	8935	6427	6523	6427
q10	9010	3781	3744	3744
q11	463	243	237	237
q12	474	237	246	237
q13	18936	2975	2985	2975
q14	279	230	218	218
q15	530	484	492	484
q16	539	378	382	378
q17	975	715	761	715
q18	8167	7515	7545	7515
q19	8407	1453	1472	1453
q20	663	332	327	327
q21	4917	3235	3268	3235
q22	420	338	333	333
Total cold run time: 121454 ms
Total hot run time: 40177 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4462	4305	4287	4287
q2	377	265	254	254
q3	2972	2730	2727	2727
q4	1911	1672	1577	1577
q5	5249	5243	5249	5243
q6	215	130	130	130
q7	2081	1725	1727	1725
q8	3196	3352	3292	3292
q9	8326	8312	8319	8312
q10	3923	3671	3671	3671
q11	581	496	491	491
q12	786	605	616	605
q13	17250	3017	3010	3010
q14	306	258	255	255
q15	530	502	476	476
q16	484	407	426	407
q17	1764	1470	1491	1470
q18	7566	7529	7557	7529
q19	1719	1600	1564	1564
q20	1987	1788	1774	1774
q21	4862	4729	4820	4729
q22	624	531	522	522
Total cold run time: 71171 ms
Total hot run time: 54050 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172242 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 450f4c6ca901ea69501a0b840e7368dc50eddfbc, data reload: false

query1	928	380	378	378
query2	6462	2568	2229	2229
query3	6662	210	210	210
query4	18977	17605	17353	17353
query5	4237	492	495	492
query6	288	171	171	171
query7	4593	310	290	290
query8	315	285	280	280
query9	8433	2382	2353	2353
query10	615	291	280	280
query11	10599	10231	9919	9919
query12	139	84	84	84
query13	1635	373	374	373
query14	8714	6541	7055	6541
query15	227	185	186	185
query16	7755	275	271	271
query17	1387	567	549	549
query18	1924	285	279	279
query19	209	160	161	160
query20	91	83	88	83
query21	210	129	130	129
query22	4165	4215	3932	3932
query23	33664	33126	33076	33076
query24	12323	2924	2790	2790
query25	699	383	422	383
query26	1879	153	150	150
query27	3082	311	313	311
query28	7881	2061	2057	2057
query29	1198	626	614	614
query30	291	154	158	154
query31	942	745	774	745
query32	99	55	55	55
query33	769	284	289	284
query34	987	478	475	475
query35	766	614	600	600
query36	1093	961	940	940
query37	291	73	76	73
query38	2862	2786	2792	2786
query39	873	801	822	801
query40	274	131	121	121
query41	53	51	51	51
query42	122	101	103	101
query43	592	557	547	547
query44	1231	733	729	729
query45	194	171	163	163
query46	1094	749	712	712
query47	1900	1787	1782	1782
query48	384	311	310	310
query49	1214	425	401	401
query50	774	388	396	388
query51	6922	6826	6752	6752
query52	97	101	92	92
query53	363	311	299	299
query54	938	445	445	445
query55	75	74	71	71
query56	290	270	274	270
query57	1153	1041	1062	1041
query58	255	249	254	249
query59	3497	3213	3031	3031
query60	304	285	282	282
query61	102	92	94	92
query62	662	451	454	451
query63	315	292	294	292
query64	9962	2251	1751	1751
query65	3167	3124	3126	3124
query66	1367	329	340	329
query67	15574	14991	15017	14991
query68	4501	542	572	542
query69	462	306	317	306
query70	1061	1057	1135	1057
query71	459	277	273	273
query72	7264	5580	5250	5250
query73	760	330	330	330
query74	5937	5499	5420	5420
query75	3596	2718	2670	2670
query76	2901	970	991	970
query77	456	298	299	298
query78	10472	9919	9737	9737
query79	2373	523	529	523
query80	2110	472	466	466
query81	581	217	222	217
query82	773	107	103	103
query83	290	175	170	170
query84	255	89	87	87
query85	2243	292	337	292
query86	500	337	286	286
query87	3265	3082	3058	3058
query88	3803	2444	2453	2444
query89	495	379	378	378
query90	2000	189	195	189
query91	130	99	105	99
query92	70	49	52	49
query93	2417	525	513	513
query94	1429	197	196	196
query95	417	319	318	318
query96	609	270	273	270
query97	3214	2995	3008	2995
query98	220	200	197	197
query99	1311	837	849	837
Total cold run time: 277513 ms
Total hot run time: 172242 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.18 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 450f4c6ca901ea69501a0b840e7368dc50eddfbc, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.05
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.47	0.49
query6	1.14	0.73	0.72
query7	0.02	0.01	0.02
query8	0.06	0.04	0.05
query9	0.55	0.49	0.49
query10	0.54	0.55	0.54
query11	0.17	0.12	0.12
query12	0.15	0.13	0.13
query13	0.59	0.60	0.59
query14	0.77	0.77	0.80
query15	0.85	0.81	0.83
query16	0.36	0.36	0.37
query17	1.02	0.95	0.98
query18	0.23	0.25	0.26
query19	1.78	1.81	1.72
query20	0.01	0.00	0.01
query21	15.45	0.74	0.67
query22	4.39	6.74	1.57
query23	18.31	1.30	1.26
query24	2.09	0.24	0.22
query25	0.15	0.08	0.08
query26	0.26	0.18	0.18
query27	0.09	0.08	0.07
query28	13.26	1.02	1.01
query29	12.63	3.25	3.24
query30	0.26	0.06	0.05
query31	2.88	0.38	0.38
query32	3.27	0.49	0.47
query33	2.88	2.90	2.94
query34	17.16	4.43	4.47
query35	4.47	4.47	4.55
query36	0.66	0.45	0.50
query37	0.20	0.16	0.16
query38	0.16	0.16	0.14
query39	0.04	0.04	0.03
query40	0.18	0.14	0.17
query41	0.10	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.75 s
Total hot run time: 30.18 s

@xzj7019
Copy link
Contributor Author

xzj7019 commented Jul 3, 2024

run feut

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 4e448b6 into apache:master Jul 3, 2024
27 of 30 checks passed
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@xzj7019 @dataroaring
[opt](nereids) recover adoptive bucket shuffle (#36784)
8b29889 
1. Recover adoptive bucket shuffle and re-using the
enable_bucket_shuffle_join to control whether to enable it, default
remains true.
2. Remove enable_bucket_shuffle_downgrade option.

Co-authored-by: zhongjian.xzj <zhongjian.xzj@zhongjianxzjdeMacBook-Pro.local>
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
@xzj7019 xzj7019 mentioned this pull request Aug 20, 2024
Merged
yiguolei pushed a commit that referenced this pull request Aug 21, 2024
@xzj7019
[opt](nereids) recover adoptive bucket shuffle (#39598)
8a562ae 
## Proposed changes

pick from #36784

Co-authored-by: xiongzhongjian <xiongzhongjian@selectdb.com>
@yiguolei
Copy link
Contributor

need merge to 2.1 because some USER need it

@yiguolei yiguolei mentioned this pull request Sep 5, 2024
Open
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@keanji-x keanji-x keanji-x approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@xzj7019 @doris-robot @yiguolei @keanji-x @morrySnow @dataroaring