Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug](backup) Add table id into convert_rowset_ids() #37089

Merged
merged 4 commits into from
Jul 15, 2024
Merged

[bug](backup) Add table id into convert_rowset_ids() #37089

merged 4 commits into from
Jul 15, 2024

Conversation

xy720
Copy link
Member

@xy720 xy720 commented Jul 1, 2024

Proposed changes

Issue Number: step 1 in version master in #37076

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Jul 1, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@xy720
Copy link
Member Author

xy720 commented Jul 1, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 40181 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2e47b9608571f8a73332fe55ae1998278159d619, data reload: false

------ Round 1 ----------------------------------
q1	18051	4858	4415	4415
q2	2644	199	190	190
q3	11524	1174	1253	1174
q4	11162	801	811	801
q5	7546	2730	2647	2647
q6	223	142	142	142
q7	978	614	621	614
q8	9614	2073	2089	2073
q9	9040	6510	6485	6485
q10	8973	3738	3698	3698
q11	455	236	232	232
q12	431	227	232	227
q13	17766	2979	2984	2979
q14	259	225	224	224
q15	524	486	484	484
q16	528	368	370	368
q17	980	776	734	734
q18	8118	7498	7386	7386
q19	4073	1483	1452	1452
q20	658	334	333	333
q21	4941	3186	3959	3186
q22	402	337	340	337
Total cold run time: 118890 ms
Total hot run time: 40181 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4440	4260	4255	4255
q2	381	273	266	266
q3	2967	2796	2711	2711
q4	1883	1640	1630	1630
q5	5245	5252	5260	5252
q6	220	128	131	128
q7	2150	1735	1754	1735
q8	3178	3357	3320	3320
q9	8344	8311	8348	8311
q10	3854	3651	3633	3633
q11	587	474	488	474
q12	786	617	626	617
q13	17721	2979	2948	2948
q14	303	269	250	250
q15	521	485	477	477
q16	473	396	433	396
q17	1785	1470	1464	1464
q18	7806	7548	7346	7346
q19	1715	1448	1519	1448
q20	1995	1788	1759	1759
q21	4820	4728	4668	4668
q22	648	551	544	544
Total cold run time: 71822 ms
Total hot run time: 53632 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 170264 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2e47b9608571f8a73332fe55ae1998278159d619, data reload: false

query1	913	384	377	377
query2	6465	2483	2440	2440
query3	6667	209	218	209
query4	19414	17577	17236	17236
query5	4253	487	489	487
query6	300	184	171	171
query7	4616	290	298	290
query8	329	320	288	288
query9	8419	2359	2366	2359
query10	620	298	278	278
query11	10481	9889	9847	9847
query12	134	83	83	83
query13	1637	352	363	352
query14	9342	7700	7023	7023
query15	236	189	185	185
query16	7758	267	265	265
query17	1897	558	509	509
query18	1819	273	263	263
query19	199	151	152	151
query20	90	79	80	79
query21	211	126	124	124
query22	4261	4057	4086	4057
query23	33570	33000	33176	33000
query24	11989	2816	2827	2816
query25	665	360	360	360
query26	1764	155	151	151
query27	2983	308	320	308
query28	7551	2062	2069	2062
query29	1088	611	605	605
query30	294	157	157	157
query31	946	710	732	710
query32	98	53	55	53
query33	761	301	305	301
query34	961	473	478	473
query35	759	621	625	621
query36	1072	972	890	890
query37	297	78	77	77
query38	2853	2707	2765	2707
query39	862	797	809	797
query40	281	122	122	122
query41	55	53	51	51
query42	120	100	102	100
query43	619	554	553	553
query44	1184	724	717	717
query45	192	163	154	154
query46	1076	719	692	692
query47	1839	1743	1757	1743
query48	364	303	299	299
query49	1189	408	414	408
query50	772	392	400	392
query51	6878	6853	6783	6783
query52	103	93	93	93
query53	361	303	295	295
query54	941	452	451	451
query55	74	75	76	75
query56	301	293	269	269
query57	1123	1038	1070	1038
query58	256	257	250	250
query59	3809	3123	3156	3123
query60	331	319	288	288
query61	92	91	110	91
query62	646	448	438	438
query63	319	295	289	289
query64	9900	2272	1812	1812
query65	3189	3127	3065	3065
query66	1342	342	346	342
query67	15309	14992	14899	14899
query68	4633	556	558	556
query69	459	308	310	308
query70	1174	1139	1150	1139
query71	387	283	288	283
query72	7360	5455	2899	2899
query73	740	333	332	332
query74	5933	5478	5467	5467
query75	3414	2649	2704	2649
query76	2760	1016	927	927
query77	469	311	309	309
query78	10315	9780	9796	9780
query79	2806	521	522	521
query80	2133	484	472	472
query81	606	228	219	219
query82	708	107	108	107
query83	311	174	176	174
query84	258	88	88	88
query85	2298	277	274	274
query86	502	312	294	294
query87	3265	3073	3071	3071
query88	4415	2386	2376	2376
query89	479	394	386	386
query90	2023	191	190	190
query91	131	97	100	97
query92	69	117	49	49
query93	2093	505	510	505
query94	1466	191	190	190
query95	404	314	319	314
query96	587	266	263	263
query97	3213	3019	3007	3007
query98	222	203	188	188
query99	1130	837	840	837
Total cold run time: 278002 ms
Total hot run time: 170264 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.3 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2e47b9608571f8a73332fe55ae1998278159d619, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.67	0.10	0.09
query5	0.50	0.48	0.49
query6	1.14	0.72	0.72
query7	0.02	0.01	0.01
query8	0.05	0.05	0.04
query9	0.55	0.47	0.49
query10	0.54	0.54	0.54
query11	0.15	0.11	0.10
query12	0.15	0.12	0.13
query13	0.60	0.58	0.58
query14	0.80	0.77	0.81
query15	0.85	0.79	0.81
query16	0.37	0.35	0.36
query17	1.04	1.02	1.00
query18	0.21	0.26	0.23
query19	1.76	1.66	1.70
query20	0.01	0.01	0.01
query21	15.46	0.76	0.65
query22	4.63	7.25	1.72
query23	18.33	1.41	1.32
query24	2.11	0.24	0.22
query25	0.16	0.08	0.08
query26	0.26	0.18	0.18
query27	0.08	0.07	0.08
query28	13.24	1.04	0.98
query29	12.62	3.29	3.26
query30	0.25	0.06	0.05
query31	2.88	0.38	0.38
query32	3.28	0.48	0.46
query33	2.94	2.98	2.93
query34	17.07	4.42	4.49
query35	4.47	4.46	4.49
query36	0.65	0.45	0.48
query37	0.19	0.15	0.16
query38	0.16	0.14	0.15
query39	0.04	0.03	0.04
query40	0.18	0.15	0.14
query41	0.08	0.04	0.05
query42	0.05	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.94 s
Total hot run time: 30.3 s

cambyzju
cambyzju reviewed Jul 2, 2024
View reviewed changes
gensrc/thrift/AgentService.thrift Outdated
@@ -309,6 +309,7 @@ struct TCloneReq {
10: optional i32 timeout_s;
11: optional Types.TReplicaId replica_id = 0
12: optional i64 partition_id
13: optional i64 table_id
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
13: optional i64 table_id
13: optional i64 table_id = 0

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

changed to -1, may be better.

@xy720
Copy link
Member Author

xy720 commented Jul 2, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 2, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 40060 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4b6ce76e8a8fb28ad2b9e90a2c42c9f2035ddf07, data reload: false

------ Round 1 ----------------------------------
q1	17995	4587	4451	4451
q2	2322	189	188	188
q3	10482	1265	1156	1156
q4	10207	793	820	793
q5	7481	2703	2674	2674
q6	223	141	137	137
q7	968	595	605	595
q8	9223	2092	2101	2092
q9	8896	6530	6476	6476
q10	8858	3701	3804	3701
q11	468	239	232	232
q12	403	235	232	232
q13	17774	2932	2973	2932
q14	258	230	217	217
q15	528	491	478	478
q16	508	371	368	368
q17	985	667	617	617
q18	8086	7412	7455	7412
q19	7179	1571	1494	1494
q20	662	316	319	316
q21	5031	3163	3893	3163
q22	390	337	336	336
Total cold run time: 118927 ms
Total hot run time: 40060 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4422	4261	4253	4253
q2	370	267	262	262
q3	3042	2748	2747	2747
q4	1825	1551	1574	1551
q5	5286	5281	5311	5281
q6	222	133	134	133
q7	2153	1767	1750	1750
q8	3238	3401	3318	3318
q9	8358	8395	8403	8395
q10	3906	3713	3673	3673
q11	595	484	489	484
q12	793	646	621	621
q13	16442	3008	2997	2997
q14	284	257	256	256
q15	513	495	479	479
q16	469	411	434	411
q17	1811	1492	1461	1461
q18	7675	7576	7379	7379
q19	1720	1458	1560	1458
q20	1982	1809	1769	1769
q21	5032	4767	4798	4767
q22	606	527	549	527
Total cold run time: 70744 ms
Total hot run time: 53972 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172559 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4b6ce76e8a8fb28ad2b9e90a2c42c9f2035ddf07, data reload: false

query1	914	374	366	366
query2	6473	2320	2362	2320
query3	6651	216	227	216
query4	19332	17490	17376	17376
query5	4180	498	485	485
query6	278	175	174	174
query7	4596	288	292	288
query8	312	301	293	293
query9	8537	2384	2348	2348
query10	605	281	284	281
query11	10506	10091	10018	10018
query12	132	85	80	80
query13	1639	359	359	359
query14	9902	7661	7477	7477
query15	237	187	186	186
query16	7827	271	263	263
query17	1926	562	517	517
query18	1945	271	271	271
query19	196	158	149	149
query20	88	80	79	79
query21	214	127	124	124
query22	4461	4142	4159	4142
query23	33587	32963	33107	32963
query24	11100	2882	2878	2878
query25	638	357	365	357
query26	1620	147	151	147
query27	2977	310	318	310
query28	7618	2067	2053	2053
query29	1011	624	604	604
query30	273	150	155	150
query31	960	724	756	724
query32	95	50	54	50
query33	764	311	291	291
query34	929	465	476	465
query35	760	614	622	614
query36	1146	936	951	936
query37	150	74	76	74
query38	2918	2750	2818	2750
query39	863	785	787	785
query40	277	125	125	125
query41	54	52	52	52
query42	120	97	99	97
query43	603	560	552	552
query44	1206	724	722	722
query45	191	166	168	166
query46	1074	742	720	720
query47	1846	1768	1748	1748
query48	368	299	292	292
query49	1170	411	406	406
query50	767	386	384	384
query51	6929	6695	6765	6695
query52	106	89	98	89
query53	361	303	293	293
query54	888	440	447	440
query55	79	77	77	77
query56	326	264	267	264
query57	1171	1035	1021	1021
query58	245	246	255	246
query59	3388	3188	3164	3164
query60	303	282	276	276
query61	93	90	92	90
query62	650	457	446	446
query63	325	287	291	287
query64	9892	2280	1748	1748
query65	3157	3096	3109	3096
query66	1389	331	338	331
query67	15621	15148	14836	14836
query68	4520	543	555	543
query69	465	299	318	299
query70	1165	1132	1059	1059
query71	412	285	271	271
query72	7080	5430	4855	4855
query73	735	327	324	324
query74	5982	5513	5467	5467
query75	3578	2678	2632	2632
query76	2829	956	928	928
query77	446	297	296	296
query78	10335	9905	9632	9632
query79	2546	515	511	511
query80	1138	470	466	466
query81	598	223	240	223
query82	775	105	107	105
query83	307	174	171	171
query84	230	86	84	84
query85	1666	282	275	275
query86	514	322	314	314
query87	3262	3091	3137	3091
query88	3644	2375	2359	2359
query89	468	376	373	373
query90	1775	186	188	186
query91	128	100	98	98
query92	71	49	50	49
query93	1815	505	502	502
query94	1188	188	186	186
query95	400	310	308	308
query96	588	272	277	272
query97	3180	3060	3003	3003
query98	221	199	200	199
query99	1247	858	852	852
Total cold run time: 274267 ms
Total hot run time: 172559 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.19 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4b6ce76e8a8fb28ad2b9e90a2c42c9f2035ddf07, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.06	0.06
query4	1.68	0.08	0.07
query5	0.50	0.49	0.49
query6	1.14	0.73	0.73
query7	0.02	0.01	0.01
query8	0.06	0.04	0.04
query9	0.56	0.49	0.49
query10	0.53	0.56	0.53
query11	0.16	0.11	0.11
query12	0.16	0.13	0.12
query13	0.59	0.58	0.58
query14	0.76	0.78	0.78
query15	0.84	0.80	0.81
query16	0.37	0.36	0.37
query17	0.97	0.95	0.94
query18	0.22	0.26	0.22
query19	1.75	1.71	1.67
query20	0.01	0.01	0.01
query21	15.45	0.77	0.67
query22	4.02	7.52	1.68
query23	18.23	1.41	1.28
query24	2.06	0.24	0.22
query25	0.16	0.09	0.09
query26	0.29	0.18	0.18
query27	0.09	0.08	0.08
query28	13.27	1.00	1.01
query29	12.63	3.31	3.26
query30	0.25	0.06	0.05
query31	2.89	0.38	0.39
query32	3.28	0.46	0.47
query33	2.88	2.94	2.86
query34	17.16	4.39	4.38
query35	4.43	4.46	4.48
query36	0.65	0.46	0.49
query37	0.18	0.15	0.16
query38	0.16	0.16	0.16
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.05	0.05	0.04
query43	0.04	0.03	0.04
Total cold run time: 109.14 s
Total hot run time: 30.19 s

@xy720
Copy link
Member Author

xy720 commented Jul 2, 2024

run p0

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jul 5, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 5, 2024

PR approved by anyone and no changes requested.

@lide-reed lide-reed self-requested a review July 15, 2024 02:08
lide-reed
lide-reed approved these changes Jul 15, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@xy720 xy720 merged commit 6ec82ec into apache:master Jul 15, 2024
26 of 31 checks passed
seawinde pushed a commit to seawinde/doris that referenced this pull request Jul 17, 2024
@xy720 @seawinde
[bug](backup) Add table id into convert_rowset_ids() (apache#37089)
8d3332c 
## Proposed changes

Issue Number: step 1 in version master in apache#37076 

<!--Describe your changes.-->
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@xy720 @dataroaring
[bug](backup) Add table id into convert_rowset_ids() (#37089)
90f295a 
## Proposed changes

Issue Number: step 1 in version master in #37076 

<!--Describe your changes.-->
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cambyzju cambyzju cambyzju approved these changes

@lide-reed lide-reed lide-reed approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@xy720 @doris-robot @lide-reed @cambyzju @dataroaring