Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](group commit) Fix prepare stmt setNull return too many filtered rows error #38262

Merged
merged 1 commit into from
Jul 24, 2024
Merged

[fix](group commit) Fix prepare stmt setNull return too many filtered rows error #38262

merged 1 commit into from
Jul 24, 2024

Conversation

mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Jul 23, 2024 •
edited
Loading

When use prepare statement and setNull to do group commit:

stmt.setString(3, "WAGERNO");
stmt.setNull(4, Types.INTEGER);

we may get too many rows filtered rows.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@mymeiyi mymeiyi changed the title [fix](group commit) Fix jdbc setNull return too many filtered rows error [fix](group commit) Fix prepare stmt setNull return too many filtered rows error Jul 23, 2024
@mymeiyi
Copy link
Contributor Author

mymeiyi commented Jul 23, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39878 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit bf6443e55eab4fcb1c0ac6fa3011681f12ecfae3, data reload: false

------ Round 1 ----------------------------------
q1	17634	4375	4258	4258
q2	2019	194	217	194
q3	10438	1170	1087	1087
q4	10189	887	818	818
q5	7541	2711	2696	2696
q6	222	138	140	138
q7	967	599	603	599
q8	9219	2059	2082	2059
q9	8540	6529	6576	6529
q10	8655	3748	3785	3748
q11	468	242	245	242
q12	396	229	228	228
q13	17766	2987	3011	2987
q14	288	237	237	237
q15	542	478	493	478
q16	481	394	380	380
q17	976	608	674	608
q18	8268	7511	7416	7416
q19	7098	1440	1440	1440
q20	711	323	327	323
q21	4866	3211	3125	3125
q22	350	298	288	288
Total cold run time: 117634 ms
Total hot run time: 39878 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4407	4251	4241	4241
q2	371	274	275	274
q3	2980	2754	2784	2754
q4	1866	1615	1599	1599
q5	5309	5284	5299	5284
q6	227	131	136	131
q7	2144	1740	1760	1740
q8	3201	3337	3318	3318
q9	8382	8371	8458	8371
q10	3856	3671	3741	3671
q11	605	481	485	481
q12	774	578	597	578
q13	17714	3029	3036	3029
q14	301	279	289	279
q15	516	486	478	478
q16	480	420	404	404
q17	1778	1482	1457	1457
q18	7661	7506	7437	7437
q19	2776	1531	1402	1402
q20	1998	1775	1773	1773
q21	5004	4798	4723	4723
q22	592	540	540	540
Total cold run time: 72942 ms
Total hot run time: 53964 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172678 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit bf6443e55eab4fcb1c0ac6fa3011681f12ecfae3, data reload: false

query1	915	364	373	364
query2	6445	1939	1799	1799
query3	6666	214	218	214
query4	28448	17587	17373	17373
query5	4207	488	483	483
query6	313	176	167	167
query7	4586	294	288	288
query8	261	203	202	202
query9	8614	2465	2407	2407
query10	428	289	261	261
query11	10725	10114	10062	10062
query12	132	84	82	82
query13	1634	383	365	365
query14	10178	7140	6985	6985
query15	231	162	169	162
query16	7817	506	495	495
query17	1135	545	544	544
query18	1918	279	270	270
query19	197	147	146	146
query20	89	80	80	80
query21	207	130	121	121
query22	4254	4032	3979	3979
query23	34277	33157	33277	33157
query24	12314	2917	2870	2870
query25	662	393	393	393
query26	1817	176	148	148
query27	2937	274	274	274
query28	7635	2029	2021	2021
query29	1123	625	607	607
query30	296	148	149	148
query31	931	758	748	748
query32	96	55	52	52
query33	771	327	323	323
query34	922	480	495	480
query35	869	759	730	730
query36	1127	920	947	920
query37	247	78	77	77
query38	2899	2744	2724	2724
query39	857	803	820	803
query40	279	120	119	119
query41	47	45	46	45
query42	114	98	131	98
query43	503	480	444	444
query44	1207	730	732	730
query45	187	167	158	158
query46	1090	712	719	712
query47	1895	1784	1798	1784
query48	365	296	289	289
query49	1206	408	414	408
query50	809	399	395	395
query51	6848	6747	6635	6635
query52	101	95	97	95
query53	351	293	287	287
query54	985	455	446	446
query55	80	72	77	72
query56	294	272	294	272
query57	1189	1080	1032	1032
query58	277	246	254	246
query59	2746	2622	2580	2580
query60	315	313	273	273
query61	97	97	96	96
query62	858	672	653	653
query63	329	289	294	289
query64	10482	2230	1705	1705
query65	3193	3089	3121	3089
query66	1411	341	336	336
query67	15509	15009	14997	14997
query68	4632	547	563	547
query69	470	351	360	351
query70	1124	1122	1171	1122
query71	398	278	283	278
query72	7331	5624	5894	5624
query73	742	344	338	338
query74	6089	5723	5704	5704
query75	3432	2709	2731	2709
query76	2751	944	902	902
query77	450	325	311	311
query78	9576	8913	8887	8887
query79	2290	513	532	513
query80	1293	468	469	468
query81	590	227	220	220
query82	763	131	135	131
query83	235	163	166	163
query84	248	96	93	93
query85	2025	318	360	318
query86	485	307	311	307
query87	3327	3098	3085	3085
query88	4205	2417	2390	2390
query89	467	381	389	381
query90	1830	199	196	196
query91	131	105	103	103
query92	69	51	50	50
query93	2492	515	526	515
query94	1259	277	299	277
query95	410	315	319	315
query96	601	274	276	274
query97	3200	3009	3026	3009
query98	213	200	206	200
query99	1546	1290	1288	1288
Total cold run time: 286621 ms
Total hot run time: 172678 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.51 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit bf6443e55eab4fcb1c0ac6fa3011681f12ecfae3, data reload: false

query1	0.05	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.07
query5	0.50	0.47	0.48
query6	1.12	0.73	0.71
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.54	0.48	0.49
query10	0.54	0.54	0.53
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.59
query14	0.77	0.79	0.83
query15	0.84	0.80	0.81
query16	0.37	0.36	0.37
query17	0.95	0.97	0.97
query18	0.23	0.22	0.22
query19	1.90	1.71	1.68
query20	0.01	0.01	0.01
query21	15.43	0.78	0.64
query22	4.69	5.92	1.86
query23	18.28	1.44	1.33
query24	2.18	0.23	0.22
query25	0.16	0.08	0.09
query26	0.30	0.21	0.21
query27	0.45	0.24	0.23
query28	13.20	1.04	1.01
query29	12.64	3.30	3.27
query30	0.26	0.06	0.06
query31	2.85	0.39	0.40
query32	3.27	0.49	0.47
query33	2.91	2.93	2.90
query34	17.08	4.32	4.35
query35	4.45	4.37	4.43
query36	0.66	0.48	0.47
query37	0.18	0.18	0.16
query38	0.16	0.15	0.15
query39	0.05	0.03	0.04
query40	0.16	0.12	0.12
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.04	0.05	0.03
Total cold run time: 110.31 s
Total hot run time: 30.51 s

dataroaring
dataroaring approved these changes Jul 24, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 24, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 1e304bd into apache:master Jul 24, 2024
30 of 32 checks passed
mymeiyi added a commit to mymeiyi/doris that referenced this pull request Jul 24, 2024
@mymeiyi
[fix](group commit) Fix prepare stmt setNull return too many filtered…
1e84979 
… rows error (apache#38262)

When use prepare statement and setNull to do group commit:
```
stmt.setString(3, "WAGERNO");
stmt.setNull(4, Types.INTEGER);
```
we may get `too many rows filtered` rows.
@mymeiyi mymeiyi mentioned this pull request Jul 24, 2024
Merged
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@mymeiyi
[fix](group commit) Fix prepare stmt setNull return too many filtered…
e23c133 
… rows error (#38262) (#38276)

Pick #38262
dataroaring pushed a commit that referenced this pull request Jul 24, 2024
@mymeiyi @dataroaring
[fix](group commit) Fix prepare stmt setNull return too many filtered…
de121f3 
… rows error (#38262)

When use prepare statement and setNull to do group commit:
```
stmt.setString(3, "WAGERNO");
stmt.setNull(4, Types.INTEGER);
```
we may get `too many rows filtered` rows.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@mymeiyi @doris-robot @Yukang-Lian @dataroaring