Use proper memory queries to rocksdb #308
Conversation
NikVolf
force-pushed
the
nv-proper-queries
branch
from
e77bfdf
to
648f4e6
Compare
NikVolf
force-pushed
the
nv-proper-queries
branch
from
256d8dd
to
3539d1f
Compare
There was a problem hiding this comment.
I wonder if we should try to do this the "RocksDB way" and use GetAggregatedIntProperty(). Also of interest is the rocksdb memory utility (https://github.com/facebook/rocksdb/blob/master/utilities/memory/memory_util.cc).
Sometimes the C API lacks some features. If that is the case here, let's try to get upstream to expose it. In previous cases they've been pretty quick to get things in.
|
@dvdplm I will look into it Approach in this pr offers rather consistent memory report from my observation, so it worth having it meanwhile |
Definitely agree we can do this incrementally, but let's make sure we open issues here or in upstream as needed so we don't lose track of what we'd rather do. Thanks! |
* master: (27 commits) update parity-util-mem (#309) Update features and feature dependencies (#307) Use proper memory queries to rocksdb (#308) Draft version updates and changelog (#299) Use custom error type for `from_hex` (#305) Fix typo. (#303) kvdb: remove KeyValueDBHandler (#304) extract common kvdb tests into a crate (#301) Expose to_hex and from_hex from impl-serde (#302) Add a top level function to parity-util-mem (#298) I/O statistic for key-value databases (#294) use for_tuples (#300) Add memory extensions for LRUCache, hashbrown (#293) Add memory stats for kvdb-s (#292) Provide #[derive(MallocSizeOf)] that is actually working (#291) Move and extend impls for locking primitives (#290) Update uint README (#288) Update README.md (#287) [ci]: remove feature flags in virtual workspace (#289) Prepare releases (#286) ...
* master: (56 commits) Remove libc completely (#333) update changelogs (#329) bump parity-util-mem to 0.4.2 (#328) remove libc feature from fixed-hash (#317) kvdb-rocksdb: release 0.4.2 (#327) kvdb-rocksdb: fix iter_from_prefix being slow (#326) MallocSizeOf for BTreeSet (#325) split off primitives (#323) travis: disable kvdb-web tests for chrome (#324) Expand const fn coverage (#319) uint: make zero const fn (#318) README: fix appveyor badge (#316) keccak-hash: switch benches to criterion (#315) update parity-util-mem (#309) Update features and feature dependencies (#307) Use proper memory queries to rocksdb (#308) Draft version updates and changelog (#299) Use custom error type for `from_hex` (#305) Fix typo. (#303) kvdb: remove KeyValueDBHandler (#304) ...
Profiling what RocksDB returns in production revealed some data:
rocksdb.estimate-table-readers-memandrocksdb.cur-size-all-mem-tablesare per-column.rocksdb.block-cache-usageis per database, even so it must be queried viadb.property_int_value_cfpassing any column (equal and shared for all columns) 🤷♀️