Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](autoinc) Fix AutoIncrementGenerator and add more logs about auto-increment column #37306

Merged
merged 1 commit into from
Jul 6, 2024
Merged

[fix](autoinc) Fix AutoIncrementGenerator and add more logs about auto-increment column #37306

merged 1 commit into from
Jul 6, 2024

Conversation

bobhan1
Copy link
Contributor

@bobhan1 bobhan1 commented Jul 4, 2024 •
edited
Loading

Proposed changes

  1. when run out batchEndId in memory, we (1) write edit log to update batchEndId, (2) update batchEndId in memory, (3) update nextId in memory
  2. add more logs about auto-increment column.
  3. remove useless lowerBound

branch-2.1-pick: #37366

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@bobhan1
Copy link
Contributor Author

bobhan1 commented Jul 4, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 39359 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 66a6c48c44b08523797e4644928bde0cb33a1d2a, data reload: false

------ Round 1 ----------------------------------
q1	17621	4360	4275	4275
q2	2007	189	186	186
q3	10473	1162	976	976
q4	10186	758	823	758
q5	7495	2618	2625	2618
q6	217	135	136	135
q7	939	599	603	599
q8	9227	2029	2062	2029
q9	8906	6455	6444	6444
q10	8886	3672	3670	3670
q11	440	239	236	236
q12	456	233	226	226
q13	18349	3004	2968	2968
q14	272	220	222	220
q15	522	497	495	495
q16	510	386	376	376
q17	960	610	723	610
q18	7919	7314	7393	7314
q19	3709	1488	1366	1366
q20	690	320	323	320
q21	4875	3928	3197	3197
q22	392	341	341	341
Total cold run time: 115051 ms
Total hot run time: 39359 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4402	4224	4234	4224
q2	364	277	264	264
q3	2948	2714	2836	2714
q4	1992	1666	1723	1666
q5	5521	5568	5461	5461
q6	215	132	142	132
q7	2165	1831	1820	1820
q8	3243	3400	3389	3389
q9	8646	8609	8722	8609
q10	4094	3897	3790	3790
q11	577	486	486	486
q12	776	629	674	629
q13	17298	3147	3157	3147
q14	288	281	265	265
q15	531	504	485	485
q16	474	427	417	417
q17	1815	1536	1501	1501
q18	8087	7882	7728	7728
q19	1791	1644	1575	1575
q20	2120	1865	1887	1865
q21	5279	5023	4981	4981
q22	639	543	570	543
Total cold run time: 73265 ms
Total hot run time: 55691 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172837 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 66a6c48c44b08523797e4644928bde0cb33a1d2a, data reload: false

query1	908	375	373	373
query2	6582	2456	2404	2404
query3	6635	207	210	207
query4	20649	17406	17364	17364
query5	3607	489	482	482
query6	262	175	181	175
query7	4585	302	300	300
query8	332	311	292	292
query9	8470	2410	2400	2400
query10	575	306	280	280
query11	10592	10076	9948	9948
query12	110	86	83	83
query13	1635	367	371	367
query14	10050	6852	7429	6852
query15	239	185	188	185
query16	7687	320	306	306
query17	1805	545	528	528
query18	1954	278	287	278
query19	196	154	156	154
query20	89	84	80	80
query21	212	127	123	123
query22	4370	4129	3978	3978
query23	33788	33489	33849	33489
query24	11060	2771	2791	2771
query25	622	402	382	382
query26	1202	166	156	156
query27	2986	339	328	328
query28	7640	2155	2127	2127
query29	891	652	637	637
query30	263	151	158	151
query31	1038	779	745	745
query32	98	53	60	53
query33	774	294	287	287
query34	997	473	483	473
query35	787	662	637	637
query36	1098	1006	976	976
query37	158	80	83	80
query38	2951	2793	2870	2793
query39	919	842	843	842
query40	230	132	131	131
query41	54	54	52	52
query42	106	101	108	101
query43	619	559	561	559
query44	1208	720	730	720
query45	202	162	160	160
query46	1060	708	733	708
query47	1850	1772	1774	1772
query48	378	290	299	290
query49	865	416	418	416
query50	758	390	395	390
query51	6934	6840	6746	6746
query52	108	90	92	90
query53	357	287	293	287
query54	887	447	469	447
query55	74	73	72	72
query56	283	266	274	266
query57	1119	1052	1030	1030
query58	267	243	256	243
query59	3464	3206	3138	3138
query60	327	275	289	275
query61	93	95	94	94
query62	598	447	437	437
query63	323	296	287	287
query64	8853	2275	1759	1759
query65	3165	3090	3124	3090
query66	764	335	340	335
query67	15321	14867	14975	14867
query68	4564	542	550	542
query69	521	435	340	340
query70	1178	1069	1129	1069
query71	413	287	285	285
query72	7749	5624	5591	5591
query73	749	336	330	330
query74	5909	5461	5486	5461
query75	3414	2684	2707	2684
query76	2720	914	908	908
query77	440	306	308	306
query78	9643	8926	8796	8796
query79	5956	536	532	532
query80	2079	509	538	509
query81	590	216	229	216
query82	1442	111	105	105
query83	324	179	172	172
query84	274	90	89	89
query85	1910	440	271	271
query86	485	301	319	301
query87	3321	3078	3088	3078
query88	4593	2363	2377	2363
query89	493	377	386	377
query90	1819	185	186	185
query91	130	114	100	100
query92	66	51	49	49
query93	5533	525	508	508
query94	1177	209	208	208
query95	404	318	323	318
query96	601	261	262	261
query97	3217	3075	2995	2995
query98	221	206	194	194
query99	1183	846	858	846
Total cold run time: 282201 ms
Total hot run time: 172837 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.49 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 66a6c48c44b08523797e4644928bde0cb33a1d2a, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.08	0.08
query5	0.49	0.48	0.49
query6	1.14	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.47	0.48
query10	0.54	0.55	0.53
query11	0.14	0.11	0.11
query12	0.14	0.12	0.12
query13	0.59	0.58	0.58
query14	0.77	0.77	0.77
query15	0.84	0.83	0.80
query16	0.37	0.36	0.38
query17	1.00	0.95	1.02
query18	0.23	0.23	0.26
query19	1.88	1.71	1.81
query20	0.02	0.01	0.01
query21	15.77	0.77	0.67
query22	4.37	7.86	1.90
query23	18.30	1.31	1.23
query24	1.96	0.26	0.22
query25	0.17	0.08	0.08
query26	0.30	0.21	0.21
query27	0.46	0.23	0.23
query28	13.25	1.01	1.00
query29	12.69	3.29	3.25
query30	0.26	0.06	0.05
query31	2.87	0.40	0.40
query32	3.25	0.48	0.46
query33	2.90	2.90	2.89
query34	17.03	4.37	4.35
query35	4.34	4.40	4.42
query36	0.65	0.49	0.47
query37	0.18	0.15	0.15
query38	0.15	0.15	0.14
query39	0.04	0.03	0.03
query40	0.15	0.13	0.13
query41	0.10	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 110.08 s
Total hot run time: 30.49 s

dataroaring
dataroaring approved these changes Jul 5, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@bobhan1 bobhan1 mentioned this pull request Jul 6, 2024
Merged
@dataroaring dataroaring merged commit 1220324 into apache:master Jul 6, 2024
29 of 33 checks passed
dataroaring pushed a commit that referenced this pull request Jul 6, 2024
@bobhan1
[branch-2.1] Picks "[fix](autoinc) Fix AutoIncrementGenerator and add…
38b3870 
… more logs about auto-increment column #37306" (#37366)

## Proposed changes

picks #37306
bobhan1 added a commit to bobhan1/doris that referenced this pull request Jul 9, 2024
@bobhan1
[fix](autoinc) Fix AutoIncrementGenerator and add more logs about aut…
996b8e4 
…o-increment column apache#37306
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@bobhan1 @dataroaring
[fix](autoinc) Fix AutoIncrementGenerator and add more logs about aut…
93f2c95 
…o-increment column (#37306)

## Proposed changes

1. when run out batchEndId in memory, we (1) write edit log to update
batchEndId, (2) update batchEndId in memory, (3) update nextId in memory
2. add more logs about auto-increment column.
3. remove useless `lowerBound`

branch-2.1-pick: #37366
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@bobhan1 @doris-robot @Yukang-Lian @dataroaring