Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat!: improve region manifest service #1268

Merged
merged 19 commits into from
Mar 31, 2023

Conversation

killme2008
Copy link
Contributor

@killme2008 killme2008 commented Mar 28, 2023

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

Main changes:

  • Adds GC task to clean manifest old checkpoints and logs.
  • Adds RepeatedTask to run the repeated task with an interval.Refactor some code.
  • Clean atomic_write_dir dir when using the local file system as storage at startup.
  • Delete manifest logs in batch by OpenDAL new API.
  • Breaking change in config:
    • Move [compaction] section to [storage.compaction].
    • Adds [storage.manifest] to configure region manifest service.

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.

Refer to a related PR or issue link (optional)

@killme2008 killme2008 added the breaking-change This pull request contains breaking changes. label Mar 28, 2023
@github-actions github-actions bot removed the breaking-change This pull request contains breaking changes. label Mar 28, 2023
@killme2008 killme2008 force-pushed the feature/tweak-region-manifest branch from 57c7849 to 86ea620 Compare March 28, 2023 15:06
@killme2008 killme2008 marked this pull request as ready for review March 29, 2023 04:08
@codecov
Copy link

codecov bot commented Mar 29, 2023

Codecov Report

Merging #1268 (7a07029) into develop (5edd2a3) will decrease coverage by 0.22%.
The diff coverage is 89.36%.

❗ Current head 7a07029 differs from pull request most recent head 6e2c2c7. Consider uploading reports for the commit 6e2c2c7 to get more accurate results

@@             Coverage Diff             @@
##           develop    #1268      +/-   ##
===========================================
- Coverage    85.65%   85.43%   -0.22%     
===========================================
  Files          495      496       +1     
  Lines        73416    74518    +1102     
===========================================
+ Hits         62882    63667     +785     
- Misses       10534    10851     +317     

@killme2008 killme2008 added the breaking-change This pull request contains breaking changes. label Mar 29, 2023
@github-actions github-actions bot removed the breaking-change This pull request contains breaking changes. label Mar 29, 2023
@killme2008 killme2008 changed the title feat: improve region manifest service feat!: improve region manifest service Mar 29, 2023
@github-actions github-actions bot added the breaking-change This pull request contains breaking changes. label Mar 29, 2023
src/common/runtime/src/error.rs Outdated Show resolved Hide resolved
src/common/runtime/src/repeated_task.rs Outdated Show resolved Hide resolved
src/common/runtime/src/repeated_task.rs Outdated Show resolved Hide resolved
killme2008 and others added 4 commits March 30, 2023 08:58
Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: LFC <bayinamine@gmail.com>
config/datanode.example.toml Show resolved Hide resolved
config/standalone.example.toml Outdated Show resolved Hide resolved
src/common/runtime/src/repeated_task.rs Show resolved Hide resolved
src/common/runtime/src/repeated_task.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/datanode/src/datanode.rs Outdated Show resolved Hide resolved
src/datanode/src/instance.rs Show resolved Hide resolved
src/storage/src/manifest/impl_.rs Show resolved Hide resolved
src/storage/src/manifest/storage.rs Outdated Show resolved Hide resolved
Co-authored-by: Yingwen <realevenyag@gmail.com>
@killme2008
Copy link
Contributor Author

@evenyag @MichaelScofield PTAL

src/cmd/src/standalone.rs Outdated Show resolved Hide resolved
@evenyag evenyag merged commit 563adba into GreptimeTeam:develop Mar 31, 2023
paomian pushed a commit to paomian/greptimedb that referenced this pull request Oct 19, 2023
* feat: try to use batch delete in ManifestLogStorage

* feat: clean temp dir when startup with file backend

* refactor: export region manifest checkpoint actions magin and refactor storage options

* feat: purge unused manifest and checkpoint files by repeat gc task

* chore: debug deleted logs

* feat: adds RepeatedTask and refactor all gc tasks

* chore: clean code

* feat: export gc_duration to manifest config

* test: assert gc works

* fix: typo

* Update src/common/runtime/src/error.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: LFC <bayinamine@gmail.com>

* fix: format

* Update src/common/runtime/src/repeated_task.rs

Co-authored-by: Yingwen <realevenyag@gmail.com>

* chore: by CR comments

* chore: by CR comments

* fix: serde default for StorageConfig

* chore: remove compaction config in StandaloneOptions

---------

Co-authored-by: LFC <bayinamine@gmail.com>
Co-authored-by: Yingwen <realevenyag@gmail.com>
@killme2008 killme2008 deleted the feature/tweak-region-manifest branch January 16, 2024 09:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking-change This pull request contains breaking changes.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants