Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](function) fix core when input not null array in foreach function #37349

Merged
merged 2 commits into from
Jul 8, 2024
Merged

[fix](function) fix core when input not null array in foreach function #37349

merged 2 commits into from
Jul 8, 2024

Conversation

Mryange
Copy link
Contributor

@Mryange Mryange commented Jul 5, 2024

Proposed changes

error code

return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);

"transform_arguments is an internal type of array. All internal types of the array are null, so an array that is not null was mistakenly treated as a null array."

core

INFO: java_cmd /mnt/disk2/yanxuecheng/java17/bin/java
INFO: jdk_version 17
start BE in local mode
F20240705 15:45:48.466758 3587508 assert_cast.h:45] Bad cast from type:doris::vectorized::ColumnArray* to doris::vectorized::ColumnNullable const*
*** Check failure stack trace: ***
    @     0x5612b1cf86a6  google::LogMessage::SendToLog()
    @     0x5612b1cf50f0  google::LogMessage::Flush()
    @     0x5612b1cf8ee9  google::LogMessageFatal::~LogMessageFatal()
    @     0x5612a3373e74  assert_cast<>()
    @     0x5612a778e60d  doris::vectorized::AggregateFunctionNullUnaryInline<>::add_batch_single_place()
    @     0x5612b0d0dba1  doris::vectorized::AggFnEvaluator::execute_single_add()
    @     0x5612b0c62b52  doris::pipeline::AggSinkLocalState::_execute_without_key()
    @     0x5612b0d0a90c  doris::pipeline::AggSinkLocalState::Executor<>::execute()
    @     0x5612b0c653df  doris::pipeline::AggSinkOperatorX::sink()
    @     0x5612b1bf27c8  doris::pipeline::PipelineTask::execute()::$_1::operator()()
    @     0x5612b1bf15c6  doris::pipeline::PipelineTask::execute()
    @     0x5612b1c0e8e7  doris::pipeline::TaskScheduler::_do_work()
    @     0x5612b1c0f72c  doris::pipeline::TaskScheduler::start()::$_0::operator()()
    @     0x5612b1c0f705  std::__invoke_impl<>()
    @     0x5612b1c0f6b5  _ZSt10__invoke_rIvRZN5doris8pipeline13TaskScheduler5startEvE3$_0JEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EES6_E4typeEOS7_DpOS8_
    @     0x5612b1c0f5bd  std::_Function_handler<>::_M_invoke()
    @     0x5612a1400905  std::function<>::operator()()
    @     0x5612a2ad6319  doris::FunctionRunnable::run()
    @     0x5612a2acf9b6  doris::ThreadPool::dispatch_thread()
    @     0x5612a2addf09  std::__invoke_impl<>()
    @     0x5612a2adde4d  std::__invoke<>()
    @     0x5612a2adde1d  _ZNSt5_BindIFMN5doris10ThreadPoolEFvvEPS1_EE6__callIvJEJLm0EEEET_OSt5tupleIJDpT0_EESt12_Index_tupleIJXspT1_EEE
    @     0x5612a2adddd6  std::_Bind<>::operator()<>()
    @     0x5612a2addda5  std::__invoke_impl<>()
    @     0x5612a2addd65  _ZSt10__invoke_rIvRSt5_BindIFMN5doris10ThreadPoolEFvvEPS2_EEJEENSt9enable_ifIX16is_invocable_r_vIT_T0_DpT1_EESA_E4typeEOSB_DpOSC_
    @     0x5612a2addbbd  std::_Function_handler<>::_M_invoke()
    @     0x5612a1400905  std::function<>::operator()()
    @     0x5612a2abecff  doris::Thread::supervise_thread()
    @     0x7f6d1c7841ca  start_thread
    @     0x7f6d1d173e73  __GI___clone
    @              (nil)  (unknown)

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@Mryange
Copy link
Contributor Author

Mryange commented Jul 5, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

clang-tidy review says "All clean, LGTM! 👍"

@morningman morningman mentioned this pull request Jul 5, 2024
Merged
@doris-robot
Copy link

TPC-H: Total hot run time: 40690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false

------ Round 1 ----------------------------------
q1	17633	4523	4310	4310
q2	2017	195	196	195
q3	10467	1191	1102	1102
q4	10186	737	833	737
q5	7469	2652	2699	2652
q6	221	138	136	136
q7	954	601	597	597
q8	9431	2061	2087	2061
q9	8801	6662	6631	6631
q10	9038	3751	3725	3725
q11	449	236	242	236
q12	491	225	223	223
q13	19313	3011	3018	3011
q14	264	231	236	231
q15	556	503	485	485
q16	511	398	375	375
q17	977	646	678	646
q18	8186	7603	7556	7556
q19	6137	1497	1343	1343
q20	653	334	314	314
q21	4971	3792	3900	3792
q22	382	340	332	332
Total cold run time: 119107 ms
Total hot run time: 40690 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4382	4374	4374
q2	374	298	284	284
q3	3290	3119	3083	3083
q4	2055	1765	1775	1765
q5	5709	5770	5629	5629
q6	242	137	134	134
q7	2444	1929	1834	1834
q8	3333	3508	3473	3473
q9	8780	8879	8761	8761
q10	4119	3839	3849	3839
q11	583	496	504	496
q12	826	629	616	616
q13	15986	3150	3202	3150
q14	321	303	259	259
q15	560	486	507	486
q16	524	454	420	420
q17	1835	1539	1522	1522
q18	8349	7966	7741	7741
q19	1784	1658	1547	1547
q20	3238	1908	1868	1868
q21	10945	4842	4816	4816
q22	638	558	563	558
Total cold run time: 80375 ms
Total hot run time: 56655 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 173309 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false

query1	917	374	390	374
query2	6465	2544	2344	2344
query3	6636	207	219	207
query4	22833	17540	17314	17314
query5	3579	480	524	480
query6	258	169	166	166
query7	4591	309	293	293
query8	290	302	284	284
query9	8718	2467	2415	2415
query10	573	296	280	280
query11	10656	10076	10130	10076
query12	114	84	81	81
query13	1639	362	364	362
query14	10152	6854	7501	6854
query15	244	184	185	184
query16	7746	294	300	294
query17	1574	552	551	551
query18	1949	283	274	274
query19	182	151	157	151
query20	90	81	79	79
query21	213	132	125	125
query22	4410	4196	3997	3997
query23	33837	33844	33527	33527
query24	10732	2852	2867	2852
query25	573	394	379	379
query26	714	161	167	161
query27	2210	318	330	318
query28	5965	2169	2146	2146
query29	872	663	633	633
query30	258	157	159	157
query31	997	766	807	766
query32	93	52	53	52
query33	658	290	296	290
query34	892	503	500	500
query35	728	628	646	628
query36	1129	988	997	988
query37	140	80	85	80
query38	3003	2861	2785	2785
query39	930	843	860	843
query40	205	130	133	130
query41	54	50	52	50
query42	120	100	107	100
query43	612	582	547	547
query44	1126	753	734	734
query45	191	165	163	163
query46	1077	749	712	712
query47	1880	1772	1798	1772
query48	385	295	306	295
query49	861	431	437	431
query50	773	387	396	387
query51	6843	6833	6719	6719
query52	105	93	97	93
query53	362	298	291	291
query54	886	469	452	452
query55	75	75	75	75
query56	316	291	298	291
query57	1112	1075	1092	1075
query58	259	256	266	256
query59	3434	3266	3267	3266
query60	314	294	298	294
query61	116	115	114	114
query62	589	457	459	457
query63	330	308	296	296
query64	9261	2327	1626	1626
query65	3160	3108	3097	3097
query66	734	326	337	326
query67	15867	14995	15120	14995
query68	5736	546	540	540
query69	707	472	373	373
query70	1206	1130	1157	1130
query71	475	283	290	283
query72	9187	5181	5412	5181
query73	787	323	319	319
query74	5997	5472	5513	5472
query75	4212	2637	2678	2637
query76	3381	1045	951	951
query77	709	322	321	321
query78	13156	9467	8963	8963
query79	6927	541	546	541
query80	1297	491	486	486
query81	595	230	226	226
query82	719	115	106	106
query83	343	173	171	171
query84	275	85	93	85
query85	708	310	303	303
query86	455	308	296	296
query87	3336	3100	3126	3100
query88	5088	2390	2355	2355
query89	506	403	402	402
query90	1964	191	192	191
query91	136	101	107	101
query92	59	49	50	49
query93	4815	524	522	522
query94	1316	210	209	209
query95	419	314	325	314
query96	621	274	266	266
query97	3215	3057	3067	3057
query98	226	204	192	192
query99	1129	833	866	833
Total cold run time: 288309 ms
Total hot run time: 173309 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 083604df5168f586dd6a5f37c7df475d9c03fba7, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.22	0.05	0.06
query4	1.66	0.09	0.09
query5	0.51	0.51	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.50	0.50
query10	0.54	0.55	0.56
query11	0.16	0.12	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.59
query14	0.78	0.77	0.78
query15	0.85	0.81	0.82
query16	0.36	0.37	0.37
query17	0.99	1.03	1.00
query18	0.22	0.27	0.25
query19	1.80	1.66	1.76
query20	0.01	0.00	0.01
query21	15.39	0.76	0.67
query22	4.62	6.38	2.26
query23	18.28	1.42	1.27
query24	2.15	0.23	0.22
query25	0.16	0.08	0.08
query26	0.30	0.22	0.21
query27	0.46	0.23	0.24
query28	13.67	1.01	1.02
query29	12.64	3.31	3.33
query30	0.26	0.07	0.05
query31	2.86	0.38	0.40
query32	3.30	0.48	0.47
query33	2.84	2.91	2.89
query34	17.15	4.35	4.36
query35	4.44	4.43	4.46
query36	0.64	0.48	0.48
query37	0.18	0.15	0.16
query38	0.16	0.14	0.14
query39	0.05	0.03	0.03
query40	0.15	0.12	0.11
query41	0.09	0.06	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.59 s
Total hot run time: 31.08 s

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 8, 2024

PR approved by anyone and no changes requested.

@zhangstar333 zhangstar333 merged commit 65bf32a into apache:master Jul 8, 2024
26 of 30 checks passed
morningman added a commit that referenced this pull request Jul 8, 2024
@morningman
[fix](case) fix struct format out files (#37350)
fec14d5 
follow up #37349
1. Some of constants are calculated in FE, it should output same format
as in BE.
    Add `FormatOptions` to control the output format of complex types.

3. Modify some of result files of test cases.
morningman added a commit to morningman/doris that referenced this pull request Jul 8, 2024
@morningman
[fix](case) fix struct format out files (apache#37350)
a0fdf62 
follow up apache#37349
1. Some of constants are calculated in FE, it should output same format
as in BE.
    Add `FormatOptions` to control the output format of complex types.

3. Modify some of result files of test cases.
Mryange added a commit to Mryange/doris that referenced this pull request Jul 15, 2024
@Mryange
[fix](function) fix core when input not null array in foreach function (
f58fd04 
apache#37349)

## Proposed changes

error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."
@Mryange Mryange mentioned this pull request Jul 15, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@Mryange
[fix](function) fix core when input not null array in foreach functio… (
e64f299 
#37798)

## Proposed changes
#37349
error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."

## Proposed changes

Issue Number: close #xxx

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@Mryange @dataroaring
[fix](function) fix core when input not null array in foreach function (
1987a17 
#37349)

## Proposed changes

error code
```C++
return creator_without_type::create<AggregateFunctionForEach>(transform_arguments, true,
                                                                      nested_function);
```
"transform_arguments is an internal type of array. All internal types of
the array are null, so an array that is not null was mistakenly treated
as a null array."
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@morningman @dataroaring
[fix](case) fix struct format out files (#37350)
8bbba2a 
follow up #37349
1. Some of constants are calculated in FE, it should output same format
as in BE.
    Add `FormatOptions` to control the output format of complex types.

3. Modify some of result files of test cases.
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhangstar333 zhangstar333 zhangstar333 approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@Mryange @doris-robot @zclllyybb @zhangstar333 @yiguolei @HappenLee @dataroaring