Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](load) fix no error url for stream load #38325

Merged
merged 2 commits into from
Jul 25, 2024
Merged

Conversation

liaoxin01
Copy link
Contributor

Previously, it was determined whether to output an error URL based on whether num_filtered_rows was greater than 0. However, in abnormal situations, the statistics of num_filtered_rows may not be accurate, resulting in no error URL output. As long as there is data in the error log file indicating filtered rows, the URL can be returned to the client without the need to determine the number of filtered rows.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@liaoxin01
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40131 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4844cee997387d483bb94f3932b431b99a5fe000, data reload: false

------ Round 1 ----------------------------------
q1	17632	4482	4264	4264
q2	2022	188	187	187
q3	10455	1142	1148	1142
q4	10191	817	829	817
q5	7526	2696	2673	2673
q6	223	139	141	139
q7	955	601	600	600
q8	9225	2106	2077	2077
q9	8742	6604	6609	6604
q10	8908	3874	3792	3792
q11	463	243	235	235
q12	474	224	229	224
q13	18822	3010	3050	3010
q14	265	229	240	229
q15	534	489	489	489
q16	505	392	383	383
q17	991	735	633	633
q18	8119	7378	7496	7378
q19	8081	1440	1414	1414
q20	691	313	318	313
q21	4932	3247	3255	3247
q22	363	281	281	281
Total cold run time: 120119 ms
Total hot run time: 40131 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4380	4250	4265	4250
q2	379	262	264	262
q3	3174	2991	2929	2929
q4	1991	1764	1703	1703
q5	5571	5514	5456	5456
q6	223	142	132	132
q7	2253	1873	1847	1847
q8	3281	3434	3452	3434
q9	8754	8798	8831	8798
q10	4126	3859	3842	3842
q11	599	503	493	493
q12	812	651	644	644
q13	16186	3225	3270	3225
q14	310	315	277	277
q15	529	495	481	481
q16	512	430	437	430
q17	1832	1539	1505	1505
q18	8220	8021	7940	7940
q19	2056	1707	1665	1665
q20	2164	1867	1864	1864
q21	5105	4958	4887	4887
q22	621	522	517	517
Total cold run time: 73078 ms
Total hot run time: 56581 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172579 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4844cee997387d483bb94f3932b431b99a5fe000, data reload: false

query1	913	377	371	371
query2	6423	1924	1790	1790
query3	6657	207	213	207
query4	27348	17592	17337	17337
query5	3624	475	481	475
query6	272	165	174	165
query7	4572	285	283	283
query8	231	187	195	187
query9	8601	2437	2411	2411
query10	449	299	264	264
query11	10989	10149	10088	10088
query12	120	87	83	83
query13	1641	381	361	361
query14	10277	7606	6017	6017
query15	212	167	169	167
query16	7740	465	458	458
query17	1293	559	564	559
query18	1970	304	282	282
query19	203	157	153	153
query20	90	86	85	85
query21	201	132	128	128
query22	4309	4167	4030	4030
query23	34292	33874	33547	33547
query24	11268	2965	2842	2842
query25	607	419	381	381
query26	1191	154	164	154
query27	2421	274	284	274
query28	7168	2086	2066	2066
query29	912	688	644	644
query30	258	149	158	149
query31	963	773	749	749
query32	96	54	72	54
query33	758	325	341	325
query34	975	483	489	483
query35	892	767	756	756
query36	1132	1007	977	977
query37	157	85	82	82
query38	2984	2860	2825	2825
query39	896	868	851	851
query40	205	118	120	118
query41	46	42	45	42
query42	115	98	101	98
query43	520	468	477	468
query44	1212	728	724	724
query45	207	185	183	183
query46	1114	738	745	738
query47	1918	1846	1843	1843
query48	376	301	308	301
query49	842	432	411	411
query50	789	391	406	391
query51	6848	6711	6590	6590
query52	107	89	90	89
query53	361	305	290	290
query54	869	468	438	438
query55	73	75	73	73
query56	282	267	265	265
query57	1141	1064	1060	1060
query58	273	258	267	258
query59	2996	2686	2911	2686
query60	306	275	269	269
query61	96	93	97	93
query62	810	647	649	647
query63	316	291	284	284
query64	9416	2262	1692	1692
query65	3176	3083	3147	3083
query66	751	332	326	326
query67	15649	15087	14920	14920
query68	4586	553	552	552
query69	604	343	343	343
query70	1156	1144	1093	1093
query71	463	289	276	276
query72	8482	5566	5902	5566
query73	749	321	323	321
query74	6126	5725	5630	5630
query75	4615	2688	2682	2682
query76	3263	930	905	905
query77	672	318	321	318
query78	9709	9167	10578	9167
query79	2321	537	527	527
query80	2405	478	472	472
query81	593	228	219	219
query82	454	137	128	128
query83	305	166	164	164
query84	267	93	82	82
query85	1709	309	304	304
query86	457	327	323	323
query87	3318	3110	3109	3109
query88	3313	2353	2363	2353
query89	482	384	393	384
query90	1944	187	187	187
query91	129	151	103	103
query92	60	53	48	48
query93	1217	515	512	512
query94	1277	297	297	297
query95	407	316	318	316
query96	583	270	274	270
query97	3208	3049	3029	3029
query98	215	202	200	200
query99	1514	1259	1279	1259
Total cold run time: 282522 ms
Total hot run time: 172579 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.29 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4844cee997387d483bb94f3932b431b99a5fe000, data reload: false

query1	0.04	0.04	0.04
query2	0.09	0.04	0.04
query3	0.22	0.05	0.06
query4	1.67	0.07	0.07
query5	0.48	0.49	0.49
query6	1.12	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.56	0.55	0.53
query11	0.15	0.12	0.12
query12	0.16	0.12	0.13
query13	0.60	0.59	0.58
query14	0.77	0.80	0.78
query15	0.85	0.82	0.82
query16	0.37	0.36	0.35
query17	1.03	0.98	0.95
query18	0.23	0.21	0.22
query19	1.81	1.75	1.69
query20	0.02	0.01	0.01
query21	15.39	0.76	0.65
query22	4.00	8.02	1.61
query23	18.32	1.43	1.20
query24	2.08	0.22	0.22
query25	0.15	0.08	0.09
query26	0.30	0.21	0.21
query27	0.45	0.23	0.23
query28	13.34	1.02	1.01
query29	12.57	3.35	3.32
query30	0.25	0.08	0.07
query31	2.85	0.40	0.39
query32	3.25	0.48	0.47
query33	2.85	2.95	2.89
query34	17.03	4.34	4.40
query35	4.47	4.43	4.51
query36	0.65	0.46	0.47
query37	0.20	0.16	0.15
query38	0.15	0.16	0.15
query39	0.05	0.04	0.04
query40	0.14	0.12	0.13
query41	0.09	0.05	0.04
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.47 s
Total hot run time: 30.29 s

@liaoxin01
Copy link
Contributor Author

run buildall

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39645 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d3922dbf441ec8fc40a638a064e1b8dfcf0c21c0, data reload: false

------ Round 1 ----------------------------------
q1	17624	4505	4300	4300
q2	2015	193	188	188
q3	10444	1216	1045	1045
q4	10189	815	822	815
q5	7615	2640	2680	2640
q6	216	140	142	140
q7	958	600	607	600
q8	9213	2044	2087	2044
q9	8752	6541	6566	6541
q10	8855	3793	3746	3746
q11	470	237	239	237
q12	444	223	217	217
q13	18861	2947	2992	2947
q14	287	231	232	231
q15	525	486	498	486
q16	498	378	374	374
q17	973	626	650	626
q18	8010	7487	7358	7358
q19	7117	1426	1303	1303
q20	688	334	331	331
q21	4868	3193	3357	3193
q22	348	288	283	283
Total cold run time: 118970 ms
Total hot run time: 39645 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4373	4253	4250	4250
q2	372	284	263	263
q3	2926	2914	2934	2914
q4	1997	1692	1732	1692
q5	5602	5532	5495	5495
q6	223	134	133	133
q7	2187	1904	1841	1841
q8	3276	3405	3402	3402
q9	8776	8771	8897	8771
q10	4081	3784	3752	3752
q11	621	506	494	494
q12	814	631	620	620
q13	15836	3192	3152	3152
q14	305	300	290	290
q15	517	503	499	499
q16	490	452	434	434
q17	1811	1565	1502	1502
q18	8156	8049	7773	7773
q19	1770	1617	1473	1473
q20	2156	1927	1872	1872
q21	5133	4971	4815	4815
q22	596	524	497	497
Total cold run time: 72018 ms
Total hot run time: 55934 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174353 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d3922dbf441ec8fc40a638a064e1b8dfcf0c21c0, data reload: false

query1	914	380	373	373
query2	6468	1933	1761	1761
query3	6645	208	218	208
query4	27296	17510	17584	17510
query5	3610	467	479	467
query6	258	166	181	166
query7	4583	293	289	289
query8	242	185	183	183
query9	8414	2417	2401	2401
query10	438	290	270	270
query11	11200	10017	10080	10017
query12	114	89	82	82
query13	1653	368	363	363
query14	10412	7908	7818	7818
query15	227	166	162	162
query16	7716	456	485	456
query17	1568	547	549	547
query18	1778	273	272	272
query19	195	142	153	142
query20	91	81	83	81
query21	204	127	131	127
query22	4519	4177	4270	4177
query23	34086	33801	33221	33221
query24	11088	2894	2905	2894
query25	655	425	397	397
query26	1109	153	153	153
query27	3032	277	285	277
query28	7619	2071	2074	2071
query29	912	638	642	638
query30	251	155	152	152
query31	974	794	735	735
query32	96	55	53	53
query33	759	350	353	350
query34	930	486	503	486
query35	881	768	784	768
query36	1157	935	962	935
query37	146	86	83	83
query38	2929	2908	2841	2841
query39	863	811	803	803
query40	204	120	120	120
query41	45	44	43	43
query42	118	97	99	97
query43	502	461	461	461
query44	1195	732	718	718
query45	218	182	181	181
query46	1105	737	745	737
query47	1889	1780	1793	1780
query48	368	293	294	293
query49	842	406	419	406
query50	782	393	394	393
query51	6811	6599	6642	6599
query52	100	99	88	88
query53	360	290	282	282
query54	903	443	438	438
query55	73	75	74	74
query56	291	303	276	276
query57	1140	1058	1041	1041
query58	248	241	253	241
query59	2743	2635	2686	2635
query60	324	282	276	276
query61	103	95	100	95
query62	794	656	659	656
query63	321	288	289	288
query64	9523	2200	1675	1675
query65	3180	3131	3094	3094
query66	770	343	338	338
query67	15425	15010	15202	15010
query68	4601	538	545	538
query69	654	432	365	365
query70	1195	1116	1172	1116
query71	412	281	287	281
query72	7080	5994	5879	5879
query73	754	323	321	321
query74	6171	5668	5743	5668
query75	3530	2712	2677	2677
query76	3358	922	887	887
query77	638	302	303	302
query78	9888	9047	8998	8998
query79	2607	531	525	525
query80	2092	477	470	470
query81	592	217	219	217
query82	805	138	136	136
query83	326	175	170	170
query84	282	92	88	88
query85	1777	318	310	310
query86	470	329	331	329
query87	3308	3105	3095	3095
query88	3811	2352	2377	2352
query89	491	382	385	382
query90	1783	197	193	193
query91	133	101	100	100
query92	64	49	48	48
query93	3288	514	511	511
query94	1151	279	303	279
query95	412	326	327	326
query96	603	282	272	272
query97	3208	3086	3020	3020
query98	219	198	194	194
query99	1581	1274	1240	1240
Total cold run time: 283693 ms
Total hot run time: 174353 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.83 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d3922dbf441ec8fc40a638a064e1b8dfcf0c21c0, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.04	0.05
query3	0.22	0.05	0.05
query4	1.68	0.07	0.07
query5	0.49	0.49	0.50
query6	1.12	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.56	0.50	0.49
query10	0.53	0.54	0.53
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.60	0.59	0.59
query14	0.77	0.77	0.79
query15	0.85	0.80	0.82
query16	0.37	0.37	0.38
query17	1.03	1.05	1.03
query18	0.23	0.21	0.22
query19	1.89	1.75	1.79
query20	0.01	0.01	0.01
query21	15.39	0.76	0.64
query22	4.36	7.23	1.91
query23	18.31	1.37	1.33
query24	2.18	0.23	0.22
query25	0.15	0.09	0.10
query26	0.32	0.22	0.21
query27	0.45	0.24	0.23
query28	13.19	1.03	1.00
query29	12.61	3.33	3.30
query30	0.25	0.06	0.05
query31	2.86	0.40	0.37
query32	3.32	0.49	0.47
query33	2.89	3.01	2.98
query34	17.20	4.36	4.39
query35	4.43	4.44	4.38
query36	0.65	0.46	0.49
query37	0.19	0.16	0.15
query38	0.16	0.14	0.15
query39	0.04	0.04	0.03
query40	0.16	0.13	0.13
query41	0.10	0.05	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.05
Total cold run time: 110.15 s
Total hot run time: 30.83 s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 25, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@gavinchou gavinchou merged commit 28292e3 into apache:master Jul 25, 2024
29 of 32 checks passed
liaoxin01 added a commit to liaoxin01/doris that referenced this pull request Jul 26, 2024
Previously, it was determined whether to output an error URL based on
whether num_filtered_rows was greater than 0. However, in abnormal
situations, the statistics of num_filtered_rows may not be accurate,
resulting in no error URL output. As long as there is data in the error
log file indicating filtered rows, the URL can be returned to the client
without the need to determine the number of filtered rows.
liaoxin01 added a commit that referenced this pull request Jul 28, 2024
dataroaring pushed a commit that referenced this pull request Jul 31, 2024
Previously, it was determined whether to output an error URL based on
whether num_filtered_rows was greater than 0. However, in abnormal
situations, the statistics of num_filtered_rows may not be accurate,
resulting in no error URL output. As long as there is data in the error
log file indicating filtered rows, the URL can be returned to the client
without the need to determine the number of filtered rows.
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Nov 12, 2024
Previously, it was determined whether to output an error URL based on
whether num_filtered_rows was greater than 0. However, in abnormal
situations, the statistics of num_filtered_rows may not be accurate,
resulting in no error URL output. As long as there is data in the error
log file indicating filtered rows, the URL can be returned to the client
without the need to determine the number of filtered rows.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants