Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add options to enable log recycle and periodical fsync #3114

Merged
merged 3 commits into from
Jan 9, 2024

Conversation

v0y4g3r
Copy link
Contributor

@v0y4g3r v0y4g3r commented Jan 7, 2024

I hereby agree to the terms of the GreptimeDB CLA

What's changed and what's your intention?

This PR adds config to enable log recycle/prefill and periodical log fsync to smooth IO utilization. This change helps both local fs and S3 mode.

Rows written per second change

  • enable_log_recycle: true
  • prefill_log_files: false
  • sync_period: 1000ms

Local FS mode

Before:
image
After:
image

S3 mode (tested on MinIO)

Before:
image
After:
image

Checklist

  • I have written the necessary rustdoc comments.
  • I have added the necessary unit tests and integration tests.
  • This PR does not require documentation updates.

Refer to a related PR or issue link (optional)

Fixes #3085

@v0y4g3r v0y4g3r added the docs-not-required This change does not impact docs. label Jan 7, 2024
@github-actions github-actions bot removed the docs-not-required This change does not impact docs. label Jan 7, 2024
@v0y4g3r v0y4g3r added A-storage Involves code in storage engines docs-not-required This change does not impact docs. C-performance Category Performance labels Jan 7, 2024
@github-actions github-actions bot removed the docs-not-required This change does not impact docs. label Jan 7, 2024
Copy link

codecov bot commented Jan 7, 2024

Codecov Report

Attention: 5 lines in your changes are missing coverage. Please review.

Comparison is base (6e860bc) 85.59% compared to head (f499125) 85.10%.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3114      +/-   ##
==========================================
- Coverage   85.59%   85.10%   -0.49%     
==========================================
  Files         815      815              
  Lines      133334   133348      +14     
==========================================
- Hits       114123   113488     -635     
- Misses      19211    19860     +649     

@v0y4g3r v0y4g3r added the docs-not-required This change does not impact docs. label Jan 8, 2024
@github-actions github-actions bot removed the docs-not-required This change does not impact docs. label Jan 8, 2024
@github-actions github-actions bot added docs-required This change requires docs update. docs-not-required This change does not impact docs. and removed docs-required This change requires docs update. labels Jan 8, 2024
src/log-store/src/raft_engine/log_store.rs Outdated Show resolved Hide resolved
Copy link
Collaborator

@fengjiachun fengjiachun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, there is some conflicts

@v0y4g3r v0y4g3r force-pushed the feat/periodical-wal-sync branch from 3ed9890 to f499125 Compare January 9, 2024 06:29
@v0y4g3r v0y4g3r enabled auto-merge January 9, 2024 06:34
@v0y4g3r v0y4g3r added this pull request to the merge queue Jan 9, 2024
Merged via the queue into GreptimeTeam:main with commit 62db28b Jan 9, 2024
22 checks passed
@v0y4g3r v0y4g3r deleted the feat/periodical-wal-sync branch January 9, 2024 06:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-storage Involves code in storage engines C-performance Category Performance docs-not-required This change does not impact docs.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Observable write stalls on high load
3 participants