-
Notifications
You must be signed in to change notification settings - Fork 480
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ignore max-db-size limit when deleting data or writing aux informations #2047
Conversation
Will this logic confuse users? |
@jihuayu There is also a risk of confusion, which we can explain in the configuration file. In addition, these delete makers will be eliminated during compaction, and the data size will decrease. If |
@caipengbo The better way is to check if it reached the max db size if it's a write command before running. And add an extra flag to identify if the write command is allowed to run even though reached the limit of db size. What you do think about this? |
That sounds better, I'll try to modify it. |
I think we can make some reserved db space, such as:
What do you think? @git-hulk @caipengbo |
Your idea is similar to what I did on our personalized version kvrocks. @jihuayu It's just that we don't want to save disk space(we reserve some disk space), we want to avoid OOM, because if we delete a lot of data, we can create a very large WAL, which caches a lot of data in memory and very easy to cause OOM(we have encountered it), and even worse, we can't restart our program, because we need to replay the WAL at startup. At this point, all unflushed data will be loaded into memory, again leading to an OOM! So we find all the places in the program where kvrocks can generate a large WAL, check it, and if it's larger than
|
Quality Gate passedThe SonarCloud Quality Gate passed, but some issues were introduced. 1 New issue |
When we delete data in kvrocks, we will write delete makers to rocksdb. When our data size reaches
max-db-size
, we can't delete data.For operations that delete data, we should ignore the
max-db-size
limit and allow the deletion to proceed, so we can clean up the data.In addition, for some aux information(written into
propagate
column family), we should also ignoremax-db-size
.