Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[enhance](mtmv)MTMV no longer generate cache when replaying logs #44283

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

zddr
Copy link
Contributor

@zddr zddr commented Nov 19, 2024

What problem does this PR solve?

When replaying logs, cache will no longer be generated because the catalog may not have been initialized or the dependent environment may not be connected, causing it to freeze here and preventing FE from starting

The cost is that after the materialized view is refreshed, the cache of the follower node will be empty, and a cache will be generated when the query is first used

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
MTMV no longer generate cache when replaying logs

Release note

MTMV no longer generate cache when replaying logs

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@zddr
Copy link
Contributor Author

zddr commented Nov 19, 2024

run buildall

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 19, 2024
Copy link
Contributor

PR approved by at least one committer and no changes requested.

Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 45241 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ba0338f77364874ccf02813683652f52c488943a, data reload: false

------ Round 1 ----------------------------------
q1	17624	7514	7315	7315
q2	2244	1170	1185	1170
q3	10075	1191	1207	1191
q4	10304	747	767	747
q5	7600	2809	2756	2756
q6	236	151	147	147
q7	999	633	613	613
q8	9406	2361	2383	2361
q9	6575	6476	6429	6429
q10	7029	2349	2316	2316
q11	472	260	259	259
q12	418	214	213	213
q13	17811	3015	3036	3015
q14	246	216	212	212
q15	584	541	513	513
q16	655	595	580	580
q17	984	539	616	539
q18	7455	6837	6664	6664
q19	1335	1081	972	972
q20	2927	2682	2705	2682
q21	3965	3205	3319	3205
q22	1394	1342	1344	1342
Total cold run time: 110338 ms
Total hot run time: 45241 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7351	7310	7429	7310
q2	337	229	232	229
q3	3077	3071	2994	2994
q4	2072	1879	1785	1785
q5	5643	5727	5712	5712
q6	228	143	136	136
q7	2236	1839	1825	1825
q8	3281	3565	3521	3521
q9	8972	8883	8926	8883
q10	3608	3612	3560	3560
q11	607	518	511	511
q12	838	632	632	632
q13	10913	3217	3322	3217
q14	311	274	272	272
q15	588	536	514	514
q16	680	644	651	644
q17	1868	1656	1608	1608
q18	8407	7704	7848	7704
q19	1695	1629	1461	1461
q20	2137	1899	1898	1898
q21	5537	5385	5427	5385
q22	629	586	576	576
Total cold run time: 71015 ms
Total hot run time: 60377 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.84 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ba0338f77364874ccf02813683652f52c488943a, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.07
query4	1.63	0.10	0.11
query5	0.42	0.43	0.40
query6	1.13	0.65	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.03
query9	0.58	0.50	0.49
query10	0.55	0.55	0.57
query11	0.14	0.11	0.10
query12	0.14	0.11	0.11
query13	0.62	0.61	0.59
query14	2.73	2.82	2.74
query15	0.91	0.83	0.84
query16	0.40	0.41	0.38
query17	0.99	1.00	1.08
query18	0.20	0.19	0.20
query19	1.91	1.79	2.08
query20	0.01	0.01	0.02
query21	15.36	0.60	0.58
query22	2.76	1.70	2.50
query23	16.88	0.93	0.67
query24	3.01	0.66	1.00
query25	0.14	0.05	0.08
query26	0.54	0.15	0.13
query27	0.04	0.05	0.04
query28	10.97	1.10	1.07
query29	12.58	3.26	3.26
query30	0.25	0.07	0.06
query31	2.85	0.37	0.38
query32	3.26	0.47	0.48
query33	3.01	2.95	3.08
query34	17.14	4.43	4.49
query35	4.55	4.53	4.50
query36	0.67	0.50	0.47
query37	0.09	0.06	0.06
query38	0.04	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.02	0.02
query43	0.04	0.03	0.04
Total cold run time: 107.27 s
Total hot run time: 31.84 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.x dev/3.0.x reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants