Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](routine load) fix enclose and escape can not set in routine load job #38402

Merged
merged 1 commit into from
Jul 30, 2024
Merged

[fix](routine load) fix enclose and escape can not set in routine load job #38402

merged 1 commit into from
Jul 30, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Jul 26, 2024

Fix enclose and escape can not set in routine load job.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Jul 26, 2024

run buildall

1 similar comment
@sollhui
Copy link
Contributor Author

sollhui commented Jul 26, 2024

run buildall

@sollhui sollhui changed the title [fix](routine-load) fix enclose and escape can not set in routine load job [fix](routine load) fix enclose and escape can not set in routine load job Jul 26, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 39314 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 024a039a7ff930cbf4da71fc3bd52da971933820, data reload: false

------ Round 1 ----------------------------------
q1	18337	4466	4360	4360
q2	2857	199	200	199
q3	11156	1171	1107	1107
q4	11364	710	772	710
q5	7912	2714	2730	2714
q6	225	141	138	138
q7	961	592	605	592
q8	9223	1919	1883	1883
q9	8907	6512	6597	6512
q10	8770	3745	3729	3729
q11	462	247	250	247
q12	397	229	219	219
q13	18771	2978	2967	2967
q14	279	241	238	238
q15	519	487	500	487
q16	496	391	386	386
q17	953	671	639	639
q18	8005	7496	7423	7423
q19	4745	1023	903	903
q20	649	334	338	334
q21	4997	3237	3391	3237
q22	356	290	291	290
Total cold run time: 120341 ms
Total hot run time: 39314 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4449	4239	4282	4239
q2	393	270	259	259
q3	3029	2757	2743	2743
q4	1890	1571	1591	1571
q5	5320	5273	5298	5273
q6	219	132	129	129
q7	2144	1763	1745	1745
q8	3201	3312	3326	3312
q9	8357	8381	8378	8378
q10	3877	3751	3661	3661
q11	582	483	502	483
q12	766	582	576	576
q13	16353	2973	2974	2973
q14	292	281	274	274
q15	507	475	483	475
q16	461	415	414	414
q17	1769	1461	1514	1461
q18	7711	7628	7384	7384
q19	1661	1443	1580	1443
q20	2000	1797	1832	1797
q21	4871	4786	4616	4616
q22	607	491	505	491
Total cold run time: 70459 ms
Total hot run time: 53697 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172369 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 024a039a7ff930cbf4da71fc3bd52da971933820, data reload: false

query1	915	373	364	364
query2	6451	1902	1895	1895
query3	6640	206	214	206
query4	28475	17632	17213	17213
query5	4240	484	497	484
query6	292	182	195	182
query7	4597	295	289	289
query8	237	196	192	192
query9	8619	2448	2434	2434
query10	450	286	269	269
query11	10602	10031	10108	10031
query12	147	89	90	89
query13	1648	380	363	363
query14	10265	7785	8151	7785
query15	232	170	166	166
query16	7742	481	468	468
query17	1573	578	538	538
query18	1976	284	274	274
query19	190	143	138	138
query20	89	84	82	82
query21	215	102	98	98
query22	4188	4030	3942	3942
query23	33901	33188	33169	33169
query24	12041	2963	2873	2873
query25	662	366	373	366
query26	1782	148	148	148
query27	2987	272	270	270
query28	7454	1997	1972	1972
query29	1098	414	409	409
query30	294	149	153	149
query31	958	726	760	726
query32	97	53	53	53
query33	795	337	322	322
query34	857	469	471	469
query35	845	734	755	734
query36	1066	917	909	909
query37	208	80	79	79
query38	2849	2797	2768	2768
query39	858	802	808	802
query40	264	115	115	115
query41	47	48	44	44
query42	120	91	95	91
query43	501	477	472	472
query44	1264	725	712	712
query45	212	180	178	178
query46	1092	729	743	729
query47	1840	1754	1769	1754
query48	371	303	305	303
query49	1196	440	407	407
query50	807	398	404	398
query51	6775	6677	6578	6578
query52	98	91	86	86
query53	253	181	176	176
query54	942	448	447	447
query55	74	74	75	74
query56	290	271	292	271
query57	1188	1058	1095	1058
query58	280	248	258	248
query59	2770	2626	2689	2626
query60	307	294	299	294
query61	98	96	94	94
query62	837	662	670	662
query63	202	180	180	180
query64	10521	2251	1694	1694
query65	3183	3114	3107	3107
query66	1375	330	343	330
query67	15535	14854	14895	14854
query68	5318	553	543	543
query69	611	414	313	313
query70	1081	1098	1073	1073
query71	450	272	260	260
query72	7343	5398	5898	5398
query73	778	327	330	327
query74	6071	5684	5663	5663
query75	3752	2670	2665	2665
query76	3245	1054	980	980
query77	699	321	303	303
query78	9877	8965	11326	8965
query79	1789	510	522	510
query80	930	491	501	491
query81	565	219	224	219
query82	499	132	131	131
query83	200	172	173	172
query84	265	80	84	80
query85	1440	327	296	296
query86	454	311	314	311
query87	3355	3078	3129	3078
query88	3262	2463	2464	2463
query89	375	278	283	278
query90	1812	197	205	197
query91	153	104	101	101
query92	64	52	48	48
query93	1074	531	513	513
query94	850	282	294	282
query95	358	258	262	258
query96	586	272	272	272
query97	3208	3044	3114	3044
query98	219	260	205	205
query99	1742	1270	1228	1228
Total cold run time: 283402 ms
Total hot run time: 172369 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.53 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 024a039a7ff930cbf4da71fc3bd52da971933820, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.05	0.04
query3	0.22	0.06	0.06
query4	1.65	0.07	0.08
query5	0.51	0.50	0.47
query6	1.14	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.05	0.04
query9	0.54	0.49	0.49
query10	0.56	0.53	0.54
query11	0.15	0.12	0.12
query12	0.14	0.11	0.12
query13	0.60	0.59	0.57
query14	0.76	0.76	0.78
query15	0.85	0.81	0.81
query16	0.37	0.36	0.37
query17	0.95	0.98	0.98
query18	0.22	0.21	0.20
query19	1.77	1.66	1.71
query20	0.02	0.01	0.01
query21	15.40	0.74	0.66
query22	4.22	7.18	2.04
query23	18.30	1.29	1.19
query24	2.14	0.22	0.23
query25	0.16	0.09	0.09
query26	0.30	0.21	0.21
query27	0.46	0.22	0.22
query28	13.29	1.02	1.00
query29	12.64	3.27	3.22
query30	0.26	0.06	0.06
query31	2.86	0.39	0.39
query32	3.29	0.48	0.47
query33	2.92	2.89	2.93
query34	17.11	4.34	4.40
query35	4.39	4.44	4.40
query36	0.65	0.50	0.48
query37	0.19	0.16	0.15
query38	0.15	0.15	0.14
query39	0.04	0.04	0.03
query40	0.15	0.13	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.05
query43	0.04	0.05	0.04
Total cold run time: 109.74 s
Total hot run time: 30.53 s

@sollhui sollhui force-pushed the support_enclose_and_escape_property branch from 024a039 to 99dfcc1 Compare July 29, 2024 02:54
@sollhui
Copy link
Contributor Author

sollhui commented Jul 29, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39177 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 99dfcc1d210fbcc275de8b6ffa2599f43f2decd4, data reload: false

------ Round 1 ----------------------------------
q1	17645	4379	4284	4284
q2	2015	202	199	199
q3	10439	1243	1106	1106
q4	10136	697	743	697
q5	7549	2724	2640	2640
q6	221	144	137	137
q7	955	590	594	590
q8	9222	1901	1927	1901
q9	8886	6544	6584	6544
q10	8826	3872	3831	3831
q11	443	250	240	240
q12	479	232	214	214
q13	17768	3001	3014	3001
q14	280	232	252	232
q15	530	480	490	480
q16	491	396	381	381
q17	966	666	668	666
q18	8100	7523	7334	7334
q19	6904	995	919	919
q20	665	335	346	335
q21	4927	3165	3880	3165
q22	347	282	281	281
Total cold run time: 117794 ms
Total hot run time: 39177 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4485	4290	4281	4281
q2	368	277	276	276
q3	3039	2893	2903	2893
q4	2060	1687	1746	1687
q5	5638	5558	5519	5519
q6	235	133	142	133
q7	2192	1877	1788	1788
q8	3274	3417	3465	3417
q9	8803	8788	8909	8788
q10	4117	3856	3812	3812
q11	595	533	491	491
q12	815	650	630	630
q13	16317	3217	3211	3211
q14	313	304	293	293
q15	532	486	477	477
q16	490	476	439	439
q17	1834	1548	1530	1530
q18	8152	7891	7840	7840
q19	1778	1747	1625	1625
q20	2194	1880	1893	1880
q21	5077	5183	4751	4751
q22	576	508	519	508
Total cold run time: 72884 ms
Total hot run time: 56269 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174030 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 99dfcc1d210fbcc275de8b6ffa2599f43f2decd4, data reload: false

query1	900	366	359	359
query2	6430	1950	1810	1810
query3	6639	211	214	211
query4	28322	17730	17541	17541
query5	3660	486	497	486
query6	283	174	162	162
query7	4584	295	293	293
query8	249	216	210	210
query9	8497	2497	2473	2473
query10	441	279	289	279
query11	11683	10030	10263	10030
query12	112	86	84	84
query13	1610	368	379	368
query14	9600	6905	7614	6905
query15	217	165	166	165
query16	7472	488	509	488
query17	1138	567	531	531
query18	1911	288	285	285
query19	189	143	137	137
query20	89	83	88	83
query21	205	98	97	97
query22	4197	3985	4010	3985
query23	34259	35092	33847	33847
query24	11230	2917	2943	2917
query25	555	409	392	392
query26	703	158	153	153
query27	2404	287	287	287
query28	6837	2132	2099	2099
query29	664	445	415	415
query30	252	152	160	152
query31	1034	763	770	763
query32	97	55	56	55
query33	760	344	324	324
query34	972	474	520	474
query35	870	768	767	767
query36	1094	959	946	946
query37	156	87	84	84
query38	2956	2919	2845	2845
query39	916	844	850	844
query40	198	123	122	122
query41	46	43	49	43
query42	106	97	99	97
query43	518	451	474	451
query44	1178	731	732	731
query45	215	176	184	176
query46	1100	736	746	736
query47	1894	1773	1754	1754
query48	359	316	294	294
query49	845	405	418	405
query50	807	418	410	410
query51	6708	6692	6676	6676
query52	101	91	89	89
query53	252	183	181	181
query54	897	445	448	445
query55	74	75	73	73
query56	298	275	276	275
query57	1138	1071	1041	1041
query58	261	277	275	275
query59	2883	2912	2601	2601
query60	304	284	291	284
query61	99	96	98	96
query62	795	659	659	659
query63	209	178	177	177
query64	9490	2257	1723	1723
query65	3176	3150	3147	3147
query66	743	342	332	332
query67	15244	15178	14854	14854
query68	4481	554	563	554
query69	543	400	338	338
query70	1132	1099	1120	1099
query71	430	274	272	272
query72	7042	5694	6405	5694
query73	765	335	337	335
query74	6111	5666	5706	5666
query75	3389	2723	2641	2641
query76	2668	957	927	927
query77	659	321	322	321
query78	11143	9926	9182	9182
query79	7903	528	528	528
query80	1819	536	521	521
query81	586	225	229	225
query82	1434	140	143	140
query83	334	186	179	179
query84	273	89	85	85
query85	1024	398	353	353
query86	353	308	309	308
query87	3275	3164	3080	3080
query88	4594	2491	2511	2491
query89	509	289	282	282
query90	1799	202	205	202
query91	140	113	118	113
query92	66	52	51	51
query93	5823	559	547	547
query94	699	312	304	304
query95	368	277	273	273
query96	630	280	278	278
query97	3219	3069	3087	3069
query98	222	205	201	201
query99	1606	1278	1278	1278
Total cold run time: 289164 ms
Total hot run time: 174030 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 99dfcc1d210fbcc275de8b6ffa2599f43f2decd4, data reload: false

query1	0.05	0.04	0.03
query2	0.08	0.04	0.04
query3	0.21	0.05	0.05
query4	1.67	0.07	0.08
query5	0.51	0.49	0.50
query6	1.12	0.73	0.73
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.49	0.49
query10	0.55	0.56	0.55
query11	0.16	0.11	0.12
query12	0.15	0.13	0.13
query13	0.61	0.58	0.59
query14	0.77	0.80	0.77
query15	0.86	0.81	0.81
query16	0.35	0.37	0.37
query17	1.00	1.01	0.97
query18	0.22	0.21	0.22
query19	1.76	1.72	1.77
query20	0.01	0.01	0.01
query21	15.42	0.77	0.67
query22	3.95	7.73	2.35
query23	18.25	1.50	1.32
query24	2.19	0.23	0.24
query25	0.16	0.09	0.09
query26	0.29	0.21	0.22
query27	0.45	0.23	0.23
query28	13.24	1.04	1.00
query29	12.59	3.36	3.34
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.26	0.47	0.47
query33	2.90	2.99	2.90
query34	16.98	4.36	4.47
query35	4.47	4.44	4.45
query36	0.65	0.50	0.48
query37	0.21	0.17	0.16
query38	0.17	0.17	0.16
query39	0.05	0.03	0.04
query40	0.14	0.12	0.12
query41	0.09	0.05	0.05
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.37 s
Total hot run time: 31.32 s

dataroaring
dataroaring approved these changes Jul 29, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 29, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

liaoxin01
liaoxin01 approved these changes Jul 30, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit 2046577 into apache:master Jul 30, 2024
28 of 30 checks passed
@sollhui sollhui mentioned this pull request Aug 4, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 4, 2024
@sollhui
[fix](routine load) fix enclose and escape can not set in routine loa…
79b07d0 
…d job (#38402) (#38825)

pick (#38402)
dataroaring pushed a commit that referenced this pull request Aug 4, 2024
@sollhui @dataroaring
[fix](routine load) fix enclose and escape can not set in routine loa…
4942677 
…d job (#38402)
@sollhui sollhui mentioned this pull request Aug 10, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 11, 2024
@sollhui
[fix](routine load) fix enclose and escape can not set in routine loa…
f2a82f6 
…d job (#38402) (#39184)

pick (#38402)
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
GoGoWen pushed a commit to GoGoWen/incubator-doris that referenced this pull request Aug 27, 2024
@sollhui
[fix](routine load) fix enclose and escape can not set in routine loa…
7aa250c 
…d job (apache#38402) (apache#39184)

pick (apache#38402)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@sollhui @doris-robot @liaoxin01 @dataroaring