-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vault performance degration due to fragmented bbolt db when using raft backend #11072
Comments
Before and after compaction. In the before case we are in the 300MB/s ballpark all the time, and after the compaction is goes to 30-40MB/s:
|
For the record with a sustained amount of approle logins in the order of 5M a day, the database starts experiencing degradation after 20 days or so, going from 20MB/s to 100MB/s, and:
It's worth noting that the actual number of secrets does not grow much in 20 days, the churn is mainly caused by token creation and expiry. |
@ncabatoff Do you think that this issue could be related to GH-11377 |
@write0nly Based on some internal investigations we've been doing, our current hypothesis is that this behavior is related to the BoltDB freelist and how it behaves as the size of the BoltDB file grows. I just merged #11895, which will be available in the next major release of Vault (1.8). I'm hopeful that these changes improve the problems you're seeing. |
@raskchanky Thank you very much! This is really great. If you have any beta builds with or without more debugging pls send me a link and I'll try it, otherwise i'll patch it and compile tomorrow. Thank you :-) |
@write0nly, based on @raskchanky's 1.8 work I'm going to treat this issue as resolved. Feel free to open a new ticket if I'm mistaken. |
Describe the bug
Vault environment using 5 nodes with vault 1.6.0 raft backend. About 1000 approle role_ids, and the very unfortunate situation of having apps that churn more than 5million approle logins a day due to batch processing, and let the background token/lease watermark expire them in bulk.
Over time vault boltdb becomes more and more fragmented, this happens to a point where the fragmentation caused boltdb to behave in a completely different way than expected by changing its normal IO write sizes from ~16k to ~5M. When that happens vault goes from doing 40MB/s writes to about 300MB/s writes and starts blocking on I/O and having a number of operation timeouts.
Normally when the database is <1G and not fragmented all is very fast, but when the db is >3.5G then all operations can be very slow due to the 5M IO size issue. See more details at the bottom of Description.
To Reproduce
We have not yet been able to reproduce it from a new database but we do have a database where the problem happens. Therefore the steps below are for now an expectation of how it could be reproduced.
Steps to reproduce the behavior:
Expected behavior
Expected behaviour is that vault will continue to have low latency and write low amounts to disk.
Environment:
Vault server configuration file(s):
Workarounds
Two possible workarounds that are very simple:
Additional context
Vault opens 2 files:
FD 8 /opt/vault/raft/vault.db about 3.6GB this is the boltdb with all vault data
FD 9 /opt/vault/raft/raft/raft.db 100MB in size, raft transactions only
during pathological behaviour we see 250MB/s writes. The I/O profile of vault is drastically different in the two scenarios.
When it is not ok, it keeps on writing lots of 5MB pwrite64s to vault.db, instead of the typical 16k~24k pwrite64s.
c8 is the counter of writes to FD8, and c9 for FD9
normal I/O, around 30MB/s:
high I/O pathological case, around 250MB/s:
The text was updated successfully, but these errors were encountered: