Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](hash join) fix numeric overflow when calculating hash table bucket size #37193

Merged
merged 1 commit into from
Jul 3, 2024
Merged

[fix](hash join) fix numeric overflow when calculating hash table bucket size #37193

merged 1 commit into from
Jul 3, 2024

Conversation

jacktengg
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@jacktengg
Copy link
Contributor Author

run buildall

github-actions[bot]
github-actions bot reviewed Jul 3, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/vec/common/hash_table/join_hash_table.h
@@ -19,6 +19,8 @@

#include <gen_cpp/PlanNodes_types.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gen_cpp/PlanNodes_types.h' file not found [clang-diagnostic-error]

#include <gen_cpp/PlanNodes_types.h>
         ^

@doris-robot
Copy link

TPC-H: Total hot run time: 40363 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5f32a69124e77f0376a015977ddd85499ba316ca, data reload: false

------ Round 1 ----------------------------------
q1	18000	4779	4410	4410
q2	2365	192	187	187
q3	10510	1158	1200	1158
q4	10195	799	862	799
q5	7483	2698	2668	2668
q6	227	137	141	137
q7	965	598	597	597
q8	9307	2117	2106	2106
q9	9082	6538	6543	6538
q10	8990	3766	3756	3756
q11	451	240	244	240
q12	423	234	240	234
q13	18858	3003	2983	2983
q14	272	233	230	230
q15	530	494	493	493
q16	531	381	378	378
q17	987	725	728	725
q18	8135	7520	7585	7520
q19	5804	1426	1338	1338
q20	668	339	340	339
q21	4860	3185	3224	3185
q22	400	345	342	342
Total cold run time: 119043 ms
Total hot run time: 40363 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4486	4331	4296	4296
q2	366	276	271	271
q3	3013	2767	2717	2717
q4	1866	1621	1636	1621
q5	5269	5306	5271	5271
q6	227	142	139	139
q7	2117	1787	1779	1779
q8	3215	3337	3364	3337
q9	8391	8385	8396	8385
q10	3884	3699	3623	3623
q11	607	493	482	482
q12	778	609	607	607
q13	17382	3002	3008	3002
q14	289	296	264	264
q15	521	483	482	482
q16	477	412	417	412
q17	1802	1501	1480	1480
q18	7877	7654	7403	7403
q19	1757	1703	1612	1612
q20	2003	1798	1788	1788
q21	4910	4671	4808	4671
q22	598	549	551	549
Total cold run time: 71835 ms
Total hot run time: 54191 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173696 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5f32a69124e77f0376a015977ddd85499ba316ca, data reload: false

query1	930	407	389	389
query2	6455	2448	2250	2250
query3	6653	206	213	206
query4	18897	17514	17270	17270
query5	4197	508	494	494
query6	259	174	164	164
query7	4594	305	294	294
query8	310	289	286	286
query9	8428	2371	2368	2368
query10	626	303	279	279
query11	10476	10015	9922	9922
query12	140	86	85	85
query13	1630	366	365	365
query14	10192	7752	7616	7616
query15	239	189	187	187
query16	7857	262	264	262
query17	1904	534	515	515
query18	1818	279	285	279
query19	200	156	158	156
query20	94	82	86	82
query21	223	131	126	126
query22	4273	3993	3987	3987
query23	33819	33354	32992	32992
query24	11764	2900	2852	2852
query25	658	366	362	362
query26	1690	159	165	159
query27	2891	320	325	320
query28	7509	2086	2069	2069
query29	1029	636	649	636
query30	284	155	151	151
query31	992	743	743	743
query32	97	51	53	51
query33	761	288	282	282
query34	958	464	485	464
query35	731	628	599	599
query36	1110	929	931	929
query37	149	76	74	74
query38	2871	2727	2722	2722
query39	866	809	809	809
query40	281	128	122	122
query41	60	58	51	51
query42	123	100	111	100
query43	608	527	567	527
query44	1203	749	745	745
query45	188	165	165	165
query46	1076	709	738	709
query47	1859	1764	1788	1764
query48	379	301	310	301
query49	1165	439	419	419
query50	771	398	390	390
query51	6966	6751	6747	6747
query52	100	101	93	93
query53	359	308	301	301
query54	940	455	452	452
query55	78	76	72	72
query56	289	256	272	256
query57	1139	1061	1046	1046
query58	252	265	266	265
query59	3536	3078	3095	3078
query60	318	293	284	284
query61	91	89	89	89
query62	653	441	456	441
query63	320	301	297	297
query64	9840	2233	1744	1744
query65	3174	3139	3154	3139
query66	1364	341	327	327
query67	15480	15172	15055	15055
query68	4586	548	559	548
query69	561	361	324	324
query70	1128	1131	1180	1131
query71	401	295	288	288
query72	7058	5604	5630	5604
query73	748	327	324	324
query74	5860	5530	5406	5406
query75	3406	2665	2687	2665
query76	2940	991	975	975
query77	705	326	328	326
query78	10321	9976	9693	9693
query79	2754	529	514	514
query80	877	496	491	491
query81	577	224	220	220
query82	759	109	105	105
query83	266	165	166	165
query84	241	87	86	86
query85	1575	274	266	266
query86	498	307	293	293
query87	3326	3065	3052	3052
query88	3931	2463	2440	2440
query89	477	381	394	381
query90	1815	190	188	188
query91	127	102	100	100
query92	57	50	50	50
query93	1914	521	518	518
query94	1210	187	188	187
query95	409	383	315	315
query96	581	266	270	266
query97	3178	2986	3008	2986
query98	223	200	194	194
query99	1110	820	831	820
Total cold run time: 274805 ms
Total hot run time: 173696 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 5f32a69124e77f0376a015977ddd85499ba316ca, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.08	0.07
query5	0.49	0.48	0.48
query6	1.13	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.05	0.04
query9	0.56	0.48	0.48
query10	0.54	0.54	0.52
query11	0.15	0.11	0.12
query12	0.16	0.12	0.12
query13	0.60	0.60	0.59
query14	0.78	0.79	0.80
query15	0.86	0.81	0.83
query16	0.36	0.37	0.37
query17	0.97	0.96	1.00
query18	0.24	0.24	0.25
query19	1.94	1.77	1.74
query20	0.01	0.01	0.01
query21	15.49	0.74	0.66
query22	4.38	6.83	1.86
query23	18.30	1.39	1.41
query24	2.08	0.23	0.24
query25	0.15	0.10	0.09
query26	0.25	0.18	0.18
query27	0.08	0.08	0.08
query28	13.26	1.02	0.99
query29	12.64	3.30	3.31
query30	0.25	0.06	0.05
query31	2.86	0.38	0.39
query32	3.29	0.48	0.47
query33	2.95	2.92	2.89
query34	17.09	4.40	4.42
query35	4.51	4.48	4.48
query36	0.66	0.50	0.51
query37	0.19	0.16	0.16
query38	0.16	0.16	0.15
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.10	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.9 s
Total hot run time: 30.73 s

HappenLee
HappenLee approved these changes Jul 3, 2024
View reviewed changes
Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 3, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 3, 2024

PR approved by anyone and no changes requested.

jacktengg added a commit to jacktengg/incubator-doris that referenced this pull request Jul 3, 2024
@jacktengg
[fix](hash join) fix numeric overflow when calculating hash table buc…
d624ded 
…ket size apache#37193
@jacktengg jacktengg mentioned this pull request Jul 3, 2024
Merged
mrhhsg
mrhhsg approved these changes Jul 3, 2024
View reviewed changes
Copy link
Member

@mrhhsg mrhhsg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jacktengg jacktengg merged commit c9884e6 into apache:master Jul 3, 2024
25 of 29 checks passed
HappenLee pushed a commit that referenced this pull request Jul 4, 2024
@jacktengg
[fix](hash join) fix numeric overflow when calculating hash table buc…
fb344b6 
…ket size #37193 (#37213)

## Proposed changes

Bp #37193
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@jacktengg @dataroaring
[fix](hash join) fix numeric overflow when calculating hash table buc…
af6690c 
…ket size (#37193)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@HappenLee HappenLee HappenLee approved these changes

@mrhhsg mrhhsg mrhhsg approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@jacktengg @doris-robot @mrhhsg @HappenLee @wm1581066 @dataroaring