Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](external catalog) Fixed non-master FE log replay during refresh #46369

Merged
merged 2 commits into from
Jan 3, 2025
Merged

[fix](external catalog) Fixed non-master FE log replay during refresh #46369

merged 2 commits into from
Jan 3, 2025

Conversation

zy-kkk
Copy link
Member

@zy-kkk zy-kkk commented Jan 3, 2025 •
edited by morningman
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

In the case of multiple FEs, non-Master FEs create Catalogs by pulling master FE logs for replay. However, when refreshing, there is no need to determine the remote name to reset the meta information. It is only determined when creating.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 3, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zy-kkk
Copy link
Member Author

zy-kkk commented Jan 3, 2025

run buildall

morningman
morningman approved these changes Jan 3, 2025
View reviewed changes
Copy link
Contributor

@morningman morningman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2025
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jan 3, 2025

PR approved by anyone and no changes requested.

@zy-kkk
Copy link
Member Author

zy-kkk commented Jan 3, 2025

run buildall

@zy-kkk zy-kkk mentioned this pull request Jan 3, 2025
Merged
@doris-robot
Copy link

TPC-H: Total hot run time: 32697 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2eed95e74acda4407513d8009d1e46185a5a1cef, data reload: false

------ Round 1 ----------------------------------
q1	17649	6113	5998	5998
q2	2050	321	174	174
q3	10393	1233	735	735
q4	10191	856	435	435
q5	7500	2181	1975	1975
q6	198	179	145	145
q7	899	731	604	604
q8	9234	1353	1190	1190
q9	5308	4918	4888	4888
q10	6750	2310	1903	1903
q11	485	280	280	280
q12	349	359	229	229
q13	17768	3726	3048	3048
q14	231	233	223	223
q15	569	500	501	500
q16	627	621	591	591
q17	558	841	321	321
q18	7064	6633	6465	6465
q19	1824	962	558	558
q20	313	309	183	183
q21	2857	2187	1952	1952
q22	359	338	300	300
Total cold run time: 103176 ms
Total hot run time: 32697 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6321	6210	6193	6193
q2	239	319	227	227
q3	2214	2663	2337	2337
q4	1377	1823	1345	1345
q5	4375	4734	4798	4734
q6	187	177	144	144
q7	2070	1974	1840	1840
q8	2590	2816	2681	2681
q9	7335	7314	7297	7297
q10	3102	3358	2834	2834
q11	598	502	500	500
q12	640	745	581	581
q13	3559	3970	3283	3283
q14	293	317	268	268
q15	582	507	501	501
q16	645	672	646	646
q17	1227	1773	1251	1251
q18	7822	7519	7459	7459
q19	831	1124	1136	1124
q20	1978	2061	1899	1899
q21	5657	5299	4915	4915
q22	621	644	589	589
Total cold run time: 54263 ms
Total hot run time: 52648 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197967 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2eed95e74acda4407513d8009d1e46185a5a1cef, data reload: false

query1	1296	976	933	933
query2	6507	2417	2367	2367
query3	11009	4745	4857	4745
query4	33509	23816	23853	23816
query5	4977	606	446	446
query6	281	207	209	207
query7	3981	525	306	306
query8	324	275	230	230
query9	9171	2642	2645	2642
query10	503	314	240	240
query11	18258	15410	15363	15363
query12	172	119	112	112
query13	1582	539	419	419
query14	10459	7883	7985	7883
query15	239	211	203	203
query16	8288	618	431	431
query17	1615	771	609	609
query18	2102	415	341	341
query19	260	188	167	167
query20	121	114	113	113
query21	204	141	115	115
query22	4762	4585	4630	4585
query23	34251	33388	33889	33388
query24	6427	2283	2314	2283
query25	454	471	394	394
query26	985	280	152	152
query27	2100	476	333	333
query28	5554	2479	2469	2469
query29	571	553	413	413
query30	215	184	151	151
query31	1019	928	913	913
query32	67	56	57	56
query33	471	364	284	284
query34	783	852	521	521
query35	798	859	788	788
query36	1055	1049	995	995
query37	119	98	75	75
query38	4308	4376	4106	4106
query39	1520	1494	1488	1488
query40	209	117	108	108
query41	45	44	43	43
query42	116	104	101	101
query43	531	541	515	515
query44	1362	835	823	823
query45	180	176	171	171
query46	877	1070	684	684
query47	1993	1999	1920	1920
query48	395	409	317	317
query49	698	501	387	387
query50	672	686	395	395
query51	7466	7336	7131	7131
query52	101	98	91	91
query53	223	253	182	182
query54	486	492	423	423
query55	86	85	81	81
query56	261	258	231	231
query57	1289	1237	1145	1145
query58	256	218	232	218
query59	3289	3276	3204	3204
query60	273	262	269	262
query61	113	109	109	109
query62	874	819	777	777
query63	228	188	194	188
query64	2755	1033	651	651
query65	3315	3275	3304	3275
query66	778	404	325	325
query67	16436	15817	15439	15439
query68	8417	707	521	521
query69	483	289	268	268
query70	1230	1163	1148	1148
query71	449	331	249	249
query72	6479	3924	3808	3808
query73	646	748	351	351
query74	10139	9292	8935	8935
query75	4737	3224	2670	2670
query76	3948	1199	792	792
query77	814	353	280	280
query78	10157	10242	9463	9463
query79	3014	792	594	594
query80	619	516	424	424
query81	530	271	226	226
query82	602	146	130	130
query83	186	161	143	143
query84	283	101	66	66
query85	793	381	317	317
query86	352	305	293	293
query87	4645	4527	4374	4374
query88	3540	2187	2171	2171
query89	427	337	301	301
query90	1853	185	184	184
query91	136	136	114	114
query92	66	55	51	51
query93	1001	873	540	540
query94	655	397	302	302
query95	333	265	255	255
query96	484	621	275	275
query97	2937	2962	2813	2813
query98	228	202	194	194
query99	1715	1596	1437	1437
Total cold run time: 297522 ms
Total hot run time: 197967 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.34 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 2eed95e74acda4407513d8009d1e46185a5a1cef, data reload: false

query1	0.03	0.03	0.05
query2	0.07	0.03	0.04
query3	0.24	0.08	0.07
query4	1.62	0.11	0.10
query5	0.42	0.42	0.41
query6	1.19	0.65	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.04
query9	0.61	0.52	0.49
query10	0.56	0.55	0.55
query11	0.15	0.10	0.10
query12	0.14	0.11	0.10
query13	0.60	0.60	0.59
query14	2.70	2.72	2.72
query15	0.90	0.83	0.83
query16	0.37	0.39	0.39
query17	1.04	1.03	0.98
query18	0.23	0.21	0.21
query19	1.95	1.84	2.02
query20	0.01	0.01	0.01
query21	15.35	0.91	0.58
query22	0.75	0.79	0.69
query23	15.27	1.43	0.59
query24	2.78	0.91	2.37
query25	0.13	0.18	0.18
query26	0.18	0.14	0.13
query27	0.05	0.06	0.04
query28	14.46	1.50	1.05
query29	12.58	4.05	3.31
query30	0.25	0.08	0.06
query31	2.84	0.63	0.37
query32	3.24	0.55	0.47
query33	3.18	3.09	3.08
query34	16.82	5.03	4.49
query35	4.50	4.44	4.47
query36	0.64	0.50	0.48
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.18	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.44 s
Total hot run time: 31.34 s

@yiguolei yiguolei merged commit ba28c94 into apache:master Jan 3, 2025
26 of 29 checks passed
yiguolei pushed a commit that referenced this pull request Jan 3, 2025
@zy-kkk
branch-2.1: [fix](external catalog) Fixed non-master FE log replay du…
e9921d7 
…ring refresh (#46373)

cherry-pick #46369
morningman pushed a commit to morningman/doris that referenced this pull request Jan 4, 2025
@zy-kkk @morningman
[fix](external catalog) Fixed non-master FE log replay during refresh (…
76f720f 
…apache#46369)



In the case of multiple FEs, non-Master FEs create Catalogs by pulling
master FE logs for replay. However, when refreshing, there is no need to
determine the remote name to reset the meta information. It is only
determined when creating.
github-actions bot pushed a commit that referenced this pull request Jan 4, 2025
@zy-kkk
[fix](external catalog) Fixed non-master FE log replay during refresh (…
c9f2070 
…#46369)



In the case of multiple FEs, non-Master FEs create Catalogs by pulling
master FE logs for replay. However, when refreshing, there is no need to
determine the remote name to reset the meta information. It is only
determined when creating.
@github-actions github-actions bot mentioned this pull request Jan 4, 2025
Closed
@morningman morningman mentioned this pull request Jan 4, 2025
Merged
morningman added a commit that referenced this pull request Jan 5, 2025
@morningman @zy-kkk
[fix](external catalog) Fixed non-master FE log replay during refresh #…
5c1d61b 
…46369 #46410 (#46418)

bp #46369 #46410

---------

Co-authored-by: zy-kkk <zhongyk10@gmail.com>
@zy-kkk zy-kkk deleted the fix_external_replay branch January 6, 2025 02:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zgxme zgxme zgxme approved these changes

@morningman morningman morningman approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@zy-kkk @Thearas @doris-robot @morningman @zgxme @yiguolei