Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](function) stddev with DecimalV2 type will result in an error #38731

Merged
merged 2 commits into from
Aug 8, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Aug 1, 2024

Proposed changes

The stddev function has a separate implementation for the DecimalV2 type, but there are issues with the implementation. Given that there is almost no existing data for DecimalV2, it will be removed here. For be, upgrading to this situation will result in an error directly.

SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented

After removing DecimalV2, parameters of type DecimalV2 will be converted to double for calculations.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Aug 1, 2024

run buildall

@github-actions github-actions bot added the doing label Aug 1, 2024
Copy link
Contributor

github-actions bot commented Aug 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 41717 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 222f057c201a808b69c9ca17db61195058e8a50b, data reload: false

------ Round 1 ----------------------------------
q1	18057	4129	4103	4103
q2	2408	205	210	205
q3	11100	1302	1352	1302
q4	10672	817	935	817
q5	7997	3020	2995	2995
q6	222	141	140	140
q7	1042	629	621	621
q8	9421	1902	1955	1902
q9	8423	6585	6576	6576
q10	8709	3818	3819	3818
q11	431	245	244	244
q12	406	234	226	226
q13	17767	2956	2943	2943
q14	268	241	246	241
q15	527	477	495	477
q16	488	403	396	396
q17	955	909	949	909
q18	7969	7325	7234	7234
q19	1381	1215	1213	1213
q20	564	310	335	310
q21	5388	4777	4768	4768
q22	359	278	277	277
Total cold run time: 114554 ms
Total hot run time: 41717 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4020	3997	3988	3988
q2	319	211	216	211
q3	2978	2986	2972	2972
q4	1907	1991	1857	1857
q5	5273	5230	5231	5230
q6	215	128	132	128
q7	2049	1634	1719	1634
q8	3166	3238	3212	3212
q9	8278	8268	8253	8253
q10	3781	3844	3839	3839
q11	545	452	458	452
q12	690	570	565	565
q13	15385	2937	2942	2937
q14	286	260	251	251
q15	516	478	477	477
q16	441	391	387	387
q17	1735	1692	1687	1687
q18	7635	7276	7127	7127
q19	1704	1663	1675	1663
q20	1963	1781	1763	1763
q21	5474	5234	5233	5233
q22	517	452	455	452
Total cold run time: 68877 ms
Total hot run time: 54318 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 167788 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 222f057c201a808b69c9ca17db61195058e8a50b, data reload: false

query1	904	377	371	371
query2	6470	1722	1698	1698
query3	6667	212	234	212
query4	18804	17519	17176	17176
query5	4252	525	530	525
query6	293	196	163	163
query7	4627	309	293	293
query8	254	199	215	199
query9	8535	2358	2344	2344
query10	433	284	276	276
query11	10392	9910	10005	9910
query12	138	89	93	89
query13	1636	382	384	382
query14	9534	7451	6332	6332
query15	204	169	164	164
query16	7118	432	457	432
query17	944	562	545	545
query18	1923	280	273	273
query19	192	139	139	139
query20	87	85	84	84
query21	204	98	96	96
query22	4331	4028	4089	4028
query23	33726	33051	33078	33051
query24	10296	3079	3028	3028
query25	677	381	380	380
query26	1770	151	148	148
query27	2946	271	274	271
query28	6925	1962	1963	1962
query29	1352	419	403	403
query30	289	149	148	148
query31	920	745	740	740
query32	101	51	55	51
query33	715	299	347	299
query34	893	476	478	476
query35	841	735	721	721
query36	993	859	847	847
query37	295	78	79	78
query38	2864	2788	2730	2730
query39	862	815	809	809
query40	281	126	115	115
query41	50	45	49	45
query42	123	102	105	102
query43	461	434	419	419
query44	1175	714	713	713
query45	204	178	176	176
query46	1107	790	805	790
query47	1777	1714	1707	1707
query48	362	290	289	289
query49	1202	433	431	431
query50	893	437	436	436
query51	6877	6750	6724	6724
query52	101	101	86	86
query53	250	180	184	180
query54	664	465	451	451
query55	80	73	75	73
query56	294	269	303	269
query57	1174	1028	1044	1028
query58	280	256	271	256
query59	2623	2361	2263	2263
query60	307	275	276	275
query61	96	93	101	93
query62	912	663	672	663
query63	216	180	181	180
query64	5927	1887	1866	1866
query65	3134	3103	3113	3103
query66	1427	340	337	337
query67	15366	14863	14822	14822
query68	4424	554	583	554
query69	441	293	301	293
query70	1119	1052	1091	1052
query71	363	303	283	283
query72	7125	2706	2462	2462
query73	763	342	376	342
query74	6028	5581	5579	5579
query75	3376	2732	2747	2732
query76	2320	1208	1265	1208
query77	524	357	329	329
query78	9479	8918	8863	8863
query79	1719	534	536	534
query80	1199	508	516	508
query81	570	228	235	228
query82	1165	132	129	129
query83	239	173	167	167
query84	270	78	80	78
query85	1293	307	305	305
query86	395	300	296	296
query87	3267	3075	3055	3055
query88	2914	2420	2424	2420
query89	390	283	292	283
query90	1753	197	205	197
query91	138	115	113	113
query92	65	54	52	52
query93	1458	612	626	612
query94	891	321	380	321
query95	376	275	269	269
query96	596	280	285	280
query97	3253	3014	3068	3014
query98	216	212	201	201
query99	1608	1279	1264	1264
Total cold run time: 261507 ms
Total hot run time: 167788 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.97 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 222f057c201a808b69c9ca17db61195058e8a50b, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.04	0.04
query3	0.22	0.06	0.05
query4	1.68	0.08	0.08
query5	0.49	0.48	0.48
query6	1.16	0.73	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.51	0.51
query10	0.55	0.57	0.57
query11	0.16	0.11	0.11
query12	0.15	0.13	0.12
query13	0.62	0.59	0.59
query14	0.78	0.79	0.81
query15	0.91	0.86	0.87
query16	0.37	0.35	0.34
query17	0.99	1.00	0.99
query18	0.22	0.20	0.21
query19	1.83	1.76	1.71
query20	0.01	0.01	0.01
query21	15.42	0.74	0.65
query22	4.14	7.66	1.22
query23	18.35	1.36	1.34
query24	2.24	0.22	0.22
query25	0.19	0.09	0.08
query26	0.33	0.21	0.22
query27	0.46	0.23	0.23
query28	13.16	1.01	0.98
query29	12.51	3.29	3.29
query30	0.26	0.05	0.05
query31	2.87	0.41	0.40
query32	3.26	0.48	0.47
query33	2.90	3.01	2.94
query34	15.51	4.25	4.25
query35	4.28	4.30	4.32
query36	0.68	0.48	0.49
query37	0.19	0.16	0.17
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.14	0.13
query41	0.11	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.18 s
Total hot run time: 29.97 s

@Mryange
Copy link
Contributor Author

Mryange commented Aug 2, 2024

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

@@ -20,20 +20,16 @@
#include <stddef.h>
#include <stdint.h>

#include <algorithm>
#include <boost/iterator/iterator_facade.hpp>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'boost/iterator/iterator_facade.hpp' file not found [clang-diagnostic-error]

#include <boost/iterator/iterator_facade.hpp>
         ^

@doris-robot
Copy link

TPC-H: Total hot run time: 42089 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1ed7c6ed37df4a84a88b16b46bbf826d0a7ab879, data reload: false

------ Round 1 ----------------------------------
q1	18028	4259	4227	4227
q2	2376	200	203	200
q3	10466	1302	1323	1302
q4	10165	846	928	846
q5	7669	3000	2988	2988
q6	223	137	137	137
q7	1069	616	615	615
q8	9429	1979	1953	1953
q9	8565	6645	6621	6621
q10	8741	3834	3852	3834
q11	424	247	247	247
q12	417	235	233	233
q13	17758	2943	2979	2943
q14	272	236	239	236
q15	535	492	491	491
q16	487	400	401	400
q17	976	935	929	929
q18	8197	7347	7295	7295
q19	1389	1217	1222	1217
q20	579	325	332	325
q21	5340	4814	4770	4770
q22	357	287	280	280
Total cold run time: 113462 ms
Total hot run time: 42089 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4047	4078	4014	4014
q2	329	229	219	219
q3	3009	3004	3002	3002
q4	1891	1893	1897	1893
q5	5264	5294	5263	5263
q6	216	131	129	129
q7	2083	1663	1664	1663
q8	3211	3299	3298	3298
q9	8379	8323	8349	8323
q10	3757	3847	3835	3835
q11	554	452	443	443
q12	702	540	547	540
q13	16735	2910	2957	2910
q14	281	265	252	252
q15	515	473	472	472
q16	438	401	393	393
q17	1728	1717	1707	1707
q18	7807	7474	7174	7174
q19	1681	1668	1670	1668
q20	1952	1749	1737	1737
q21	5440	5144	5261	5144
q22	510	451	468	451
Total cold run time: 70529 ms
Total hot run time: 54530 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169775 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1ed7c6ed37df4a84a88b16b46bbf826d0a7ab879, data reload: false

query1	926	381	369	369
query2	6481	1804	1798	1798
query3	6687	216	220	216
query4	19595	17593	17302	17302
query5	4315	534	532	532
query6	301	165	160	160
query7	4608	304	294	294
query8	262	207	208	207
query9	8485	2391	2380	2380
query10	459	291	271	271
query11	10719	10096	10149	10096
query12	142	92	87	87
query13	1657	390	392	390
query14	8807	7181	6889	6889
query15	209	167	170	167
query16	7109	459	432	432
query17	1005	584	582	582
query18	1919	296	285	285
query19	204	148	154	148
query20	94	90	88	88
query21	219	100	100	100
query22	4329	4007	4249	4007
query23	33999	32949	33105	32949
query24	10284	3074	3059	3059
query25	711	413	430	413
query26	1821	155	158	155
query27	2995	282	284	282
query28	6988	2004	1982	1982
query29	1381	448	439	439
query30	300	155	151	151
query31	949	775	763	763
query32	105	60	58	58
query33	723	351	330	330
query34	918	487	507	487
query35	865	706	718	706
query36	1007	879	853	853
query37	298	87	83	83
query38	2863	2761	2778	2761
query39	900	831	848	831
query40	286	122	121	121
query41	51	47	47	47
query42	132	100	107	100
query43	472	450	452	450
query44	1242	753	757	753
query45	211	179	175	175
query46	1103	826	802	802
query47	1780	1709	1700	1700
query48	382	307	311	307
query49	1200	452	451	451
query50	917	450	464	450
query51	6725	6806	6715	6715
query52	106	92	97	92
query53	263	187	191	187
query54	672	472	484	472
query55	78	79	79	79
query56	381	268	297	268
query57	1133	1043	1044	1043
query58	285	282	285	282
query59	2739	2506	2418	2418
query60	321	283	290	283
query61	96	91	110	91
query62	917	671	671	671
query63	220	190	193	190
query64	5855	1969	1929	1929
query65	3186	3209	3113	3113
query66	1433	334	338	334
query67	15429	14822	14828	14822
query68	4456	572	588	572
query69	530	306	302	302
query70	1131	1029	1081	1029
query71	457	279	293	279
query72	7963	2679	2491	2491
query73	765	336	347	336
query74	6076	5716	5606	5606
query75	3873	2771	2774	2771
query76	2950	1176	1243	1176
query77	694	345	318	318
query78	9483	8913	8874	8874
query79	2922	549	552	549
query80	1460	506	519	506
query81	565	228	230	228
query82	815	140	137	137
query83	357	172	177	172
query84	283	86	81	81
query85	2047	312	304	304
query86	450	299	282	282
query87	3321	3132	3123	3123
query88	4057	2546	2576	2546
query89	416	303	286	286
query90	2048	201	200	200
query91	132	103	109	103
query92	64	50	53	50
query93	3188	623	626	623
query94	1040	291	296	291
query95	383	274	280	274
query96	620	290	292	290
query97	3259	3068	3091	3068
query98	213	196	200	196
query99	1612	1302	1265	1265
Total cold run time: 270884 ms
Total hot run time: 169775 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.46 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1ed7c6ed37df4a84a88b16b46bbf826d0a7ab879, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.03	0.04
query3	0.22	0.05	0.04
query4	1.68	0.08	0.08
query5	0.49	0.48	0.48
query6	1.16	0.73	0.71
query7	0.02	0.01	0.02
query8	0.05	0.05	0.05
query9	0.59	0.51	0.51
query10	0.57	0.56	0.55
query11	0.15	0.12	0.11
query12	0.15	0.13	0.13
query13	0.62	0.62	0.60
query14	0.77	0.81	0.81
query15	0.90	0.87	0.86
query16	0.35	0.36	0.36
query17	1.03	1.01	0.97
query18	0.22	0.20	0.21
query19	1.81	1.74	1.78
query20	0.02	0.01	0.01
query21	15.39	0.79	0.66
query22	4.27	7.06	1.77
query23	17.98	1.30	1.22
query24	2.27	0.24	0.22
query25	0.19	0.08	0.08
query26	0.32	0.22	0.21
query27	0.46	0.23	0.23
query28	13.17	1.00	0.97
query29	12.50	3.32	3.33
query30	0.26	0.05	0.06
query31	2.86	0.41	0.40
query32	3.25	0.49	0.48
query33	2.93	2.96	2.93
query34	15.47	4.25	4.26
query35	4.31	4.29	4.30
query36	0.68	0.48	0.48
query37	0.20	0.17	0.17
query38	0.17	0.16	0.16
query39	0.05	0.04	0.03
query40	0.16	0.14	0.13
query41	0.09	0.04	0.04
query42	0.06	0.04	0.05
query43	0.05	0.04	0.04
Total cold run time: 108.02 s
Total hot run time: 30.46 s

Copy link
Contributor

@HappenLee HappenLee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 5, 2024
@HappenLee HappenLee added dev/2.1.x and removed approved Indicates a PR has been approved by one committer. labels Aug 5, 2024
Copy link
Contributor

github-actions bot commented Aug 5, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 6, 2024
Copy link
Contributor

github-actions bot commented Aug 6, 2024

PR approved by at least one committer and no changes requested.

@HappenLee HappenLee merged commit 7c58c71 into apache:master Aug 8, 2024
29 of 31 checks passed
Mryange added a commit to Mryange/doris that referenced this pull request Aug 8, 2024
…pache#38731)

The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
Mryange added a commit to Mryange/doris that referenced this pull request Aug 8, 2024
…pache#38731)

The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
dataroaring pushed a commit that referenced this pull request Aug 8, 2024
…38731)

The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
yiguolei pushed a commit that referenced this pull request Aug 8, 2024
#39072)

…38731)

#38731
The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
…pache#38731)

The stddev function has a separate implementation for the DecimalV2
type, but there are issues with the implementation. Given that there is
almost no existing data for DecimalV2, it will be removed here. For be,
upgrading to this situation will result in an error directly.
```
SELECT STDDEV(data) FROM DECIMALV2_10_0_DATA;
ERROR 1105 (HY000): errCode = 2, detailMessage = (127.0.0.1)[INTERNAL_ERROR]Agg Function stddev(decimal(10,0)) is not implemented
```
After removing DecimalV2, parameters of type DecimalV2 will be converted
to double for calculations.
@yiguolei yiguolei mentioned this pull request Sep 5, 2024
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants