Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix RocksDB can't auto resume after disk quota exceeded error #628

Merged
merged 4 commits into from
Jun 10, 2022

Conversation

caipengbo
Copy link
Contributor

Fixed: #627

In #229, the issue where RocksDB could not recover from the no Space background error was fixed. This problem RocksDB at facebook/rocksdb#8376 has been repaired, but the issue has not been thoroughly solved, The same problem will still occur when an EDQUOT Disk Quota Exceeded error is encountered (see the detailed in facebook/rocksdb#10134).

RocksDB cannot recover from this problem and must be restarted. This problem is more likely to occur when kvrocks is deployed in container.

In order to handle all versions of RocksDB, we manually resume DB when we encounter two retryable io errors: No space left on device and Disk Quota Exceeded.

For the Disk Quota Exceeded error, RocksDB did not expose a friendly interface, so I did a string match.

@caipengbo caipengbo requested review from git-hulk and ShooterIT June 9, 2022 11:21
src/event_listener.cc Outdated Show resolved Hide resolved
src/event_listener.cc Outdated Show resolved Hide resolved
@caipengbo caipengbo requested a review from git-hulk June 9, 2022 12:10
src/event_listener.cc Outdated Show resolved Hide resolved
src/event_listener.cc Outdated Show resolved Hide resolved
ShooterIT
ShooterIT previously approved these changes Jun 10, 2022
src/event_listener.cc Outdated Show resolved Hide resolved
@ShooterIT
Copy link
Member

LGTM, it seems, in cloud environments, we may encounter this error.

@ShooterIT ShooterIT merged commit 5cf450f into apache:unstable Jun 10, 2022
@caipengbo caipengbo deleted the cannot-resume-when-edquot branch June 10, 2022 07:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

RocksDB can't auto resume after disk quota exceeded error
3 participants