Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RockDB Subcommand for printing usage per column family #6185

Merged
merged 8 commits into from
Dec 5, 2023

Conversation

siladu
Copy link
Contributor

@siladu siladu commented Nov 20, 2023

Pretty much a lift and shift of @ahamlat's https://github.com/ahamlat/RocksdDB-Column-Families-Size

➜  besu git:(rocksdb-usage-subcommand) $BESU storage x-rocksdb usage --help
Usage: besu storage x-rocksdb usage [-hV]
Prints disk usage
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
  
➜  besu git:(rocksdb-usage-subcommand) $BESU storage x-rocksdb --help
Usage: besu storage x-rocksdb [-hV] [COMMAND]
Print RocksDB information
  -h, --help      Show this help message and exit.
  -V, --version   Print version information and exit.
Commands:
  usage  Prints disk usage
$BESU --data-path=/tmp/besu storage x-rocksdb usage
****** Column family 'BLOCKCHAIN' size: 7 KiB ******
Number of keys : 70
Number of live snapshots : 0
Total size of SST Files : 4 KiB
Size of live SST Filess : 4 KiB
Column family size (with getColumnFamilyMetaData) : 4 KiB

****** Column family 'ACCOUNT_INFO_STATE' size: 1 KiB ******
Number of keys : 2
Number of live snapshots : 0
Total size of SST Files : 1 KiB
Size of live SST Filess : 1 KiB
Column family size (with getColumnFamilyMetaData) : 1 KiB

****** Column family 'TRIE_BRANCH_STORAGE' size: 1 KiB ******
Number of keys : 7
Number of live snapshots : 0
Total size of SST Files : 2 KiB
Size of live SST Filess : 2 KiB
Column family size (with getColumnFamilyMetaData) : 2 KiB

****** Column family 'TRIE_LOG_STORAGE' size: 5 KiB ******
Number of keys : 103
Number of live snapshots : 0
Total size of SST Files : 7 KiB
Size of live SST Filess : 7 KiB
Column family size (with getColumnFamilyMetaData) : 7 KiB

****** Column family 'VARIABLES' size: 1 KiB ******
Number of keys : 6
Number of live snapshots : 0
Total size of SST Files : 2 KiB
Size of live SST Filess : 2 KiB
Column family size (with getColumnFamilyMetaData) : 2 KiB

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
Copy link

github-actions bot commented Nov 20, 2023

  • I thought about documentation and added the doc-change-required label to this PR if updates are required.
  • I thought about the changelog and included a changelog update if required.
  • If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

Signed-off-by: Simon Dudley <simon.dudley@consensys.net>
@siladu siladu added the TeamGroot GH issues worked on by Groot Team label Nov 20, 2023
@siladu siladu mentioned this pull request Nov 21, 2023
@gfukushima gfukushima self-assigned this Nov 30, 2023
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
@gfukushima
Copy link
Contributor

gfukushima commented Dec 1, 2023

I've changed it to follow the MD format suggested by @jframe. Also have removed the experimental prefix.

@gfukushima gfukushima changed the title Add Experimental RockDB Subcommand for printing usage per column family Add RockDB Subcommand for printing usage per column family Dec 1, 2023
Copy link
Contributor

@fab-10 fab-10 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like it, there are still a couple of not resolved comment

Comment on lines +79 to +95
static void printTableHeader(final PrintWriter out) {
out.format(
"| Column Family | Keys | Column Size | SST Files Size |\n");
out.format(
"|--------------------------------|-----------------|--------------|-----------------|\n");
}

static void printLine(
final PrintWriter out,
final String cfName,
final String keys,
final String columnSize,
final String sstFilesSize) {
final String format = "| %-30s | %-15s | %-12s | %-15s |\n";
out.format(format, cfName, keys, columnSize, sstFilesSize);
}
}
Copy link
Contributor

@garyschulte garyschulte Dec 1, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using https://github.com/vdmeer/asciitable this could be replaced by:

AsciiTable at = new AsciiTable();
at.addRow("Column Family", "Keys", "Column Size", "SST FIles Size")
at.addRule()

per cf: at.addRow(cfName, keys, columnSize, sstFilesSize);

and finally out.print(at.render());

non-blocking feedback. Using this lib might make the output a bit more flexible for different sized terminals and such

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spiked on this and didn't like the result tbh. It is sort of terminal-friendly but it did break the first column in my embedded terminal on IntelliJ. There's some customization we could use like defining the width of columns etc, but I think this might be an overkill. I'd rather have this MD formatted at the cost of having 3 output.format().

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
@gfukushima gfukushima enabled auto-merge (squash) December 4, 2023 23:53
@gfukushima gfukushima merged commit e3db18e into hyperledger:main Dec 5, 2023
18 checks passed
jflo pushed a commit to jflo/besu that referenced this pull request Dec 12, 2023
…er#6185)

* Add RockDB Subcommand for printing usage per column family
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

* changed output to follow a MD table notation.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>
jflo added a commit that referenced this pull request Dec 12, 2023
* New cli options to limit rewards return by eth_feeHistory  (#6202)
* [#5851] Add error messages on authentication failures with username and password (#6212)
* Add a constant for the 'password'
* Add error messages on authentication failures with username and password

Signed-off-by: Justin Florentine <justin+github@florentine.us>
Signed-off-by: jflo <justin+github@florentine.us>
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>



Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Add test to check empty login and check response in body is not empty

Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Correct format (spotless)

Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Update ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/JsonRpcHttpServiceLoginTest.java

Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Update ethereum/api/src/test/java/org/hyperledger/besu/ethereum/api/jsonrpc/JsonRpcHttpServiceLoginTest.java

Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Update JsonRpcHttpServiceLoginTest.java

use containsIgnoringCase

Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

* Add a CHANGELOG entry for PR 6212

Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>

---------

Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: jflo <justin+github@florentine.us>

* Add RockDB Subcommand for printing usage per column family (#6185)

* Add RockDB Subcommand for printing usage per column family
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

* changed output to follow a MD table notation.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Deprecation warning if Forest pruning is enabled (#6230)

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: jflo <justin+github@florentine.us>

* Fix the annoying "Errors occurred while build effective model" during builds (#6241)

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: jflo <justin+github@florentine.us>

* Run ATs sequentially (#6244)

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* fix: double calls to trace{Start,End}Transaction (#6247)

Signed-off-by: delehef <franklin.delehelle@odena.eu>
Signed-off-by: jflo <justin+github@florentine.us>

* migrate to junit5 (#6234)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* fixes for problems discovered in main (#6248)

Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Pki - migrate to junit 5 (#6235)

* migrate to junit5

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* fix: double calls to trace{Start,End}Transaction (#6247)

Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu>

* migrate to junit5 (#6234)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* fixes for problems discovered in main (#6248)

Signed-off-by: garyschulte <garyschulte@gmail.com>

* fixed test comparing size of collection

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: delehef <franklin.delehelle@odena.eu>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* junit 5 ftw (#6253)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* removed unnecessary use of static temp dir  (#6251)

* don't use static tempdir

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Remove parallelism usage from mainnet AT (#6252)

* Remove parallelism usage from mainnet AT

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* Increase parallelism usage from mainnet AT

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

* Add the split command back

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* fix log params (#6254)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com>
Signed-off-by: jflo <justin+github@florentine.us>

* add dependency on jar task (#6255)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Fix and test that the BlockAwareOperationTracer methods are invoked the correct number of times (#6259)

* Test that the BlockAwareOperationTracer are invoked the correct number of times
* Remove redundant calls to traceEndBlock

Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: jflo <justin+github@florentine.us>

* [RPC] Use apiConfiguration to limit gasPrice in eth_getGasPrice (#6243)

Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: jflo <justin+github@florentine.us>

* log bootnodes and static nodes list at debug level (#6273)

* log bootnodes and static nodes list at debug level

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* log if zero bootnodes

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

* null safeguards

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* junit 5 (#6256)

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Non bft group ats junit 5 (#6260)

* migrate to junit 5

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Txparse subcommand implementation (#6268)

* txparse subcommand

Signed-off-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* [MINOR] migrate remaining Crypto tests to junit 5 (#6280)

* update crypto tests to junit5

* fixed temp file

* removed vintage junit dep

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>

---------

Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* add a fix to load correctly the storage trie in the Bonsai WorldState (#6205)

revert some modification that was made to pass tests #5686 and fix this tests by loading the storage with EMPTY_TRIE_HASH if we detect that it has been cleared before pushing the new slots after recreation.

---------

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: matkt <karim.t2am@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* Sequenced pool synonym for legacy pool (#6274)

* Sequenced pool synonym for legacy pool

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

* Class rename

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

* Spotless fixes

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

* Add SEQUENCED to config overview test

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>

* Update CHANGELOG.md

Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: Matt Whitehead <matthew1001@hotmail.com>

* add a fix to load correctly the storage trie in the Bonsai WorldState (#6205)

revert some modification that was made to pass tests #5686 and fix this tests by loading the storage with EMPTY_TRIE_HASH if we detect that it has been cleared before pushing the new slots after recreation.

---------

Signed-off-by: Karim TAAM <karim.t2am@gmail.com>

---------

Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Signed-off-by: Matt Whitehead <matthew1001@hotmail.com>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: matkt <karim.t2am@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* ETC mainnet 'Spiral' activation block (#6267)

* Set ENR tree for DNS discovery for ETC mainnet network

Signed-off-by: Diego López León <dieguitoll@gmail.com>

* Set activation block number for ECIP-1109 on ETC mainnet

Signed-off-by: Diego López León <dieguitoll@gmail.com>

---------

Signed-off-by: Diego López León <dieguitoll@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>

* uprev to version 23.10.3-RC3

Signed-off-by: jflo <justin+github@florentine.us>

---------

Signed-off-by: Justin Florentine <justin+github@florentine.us>
Signed-off-by: jflo <justin+github@florentine.us>
Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Signed-off-by: David Lutzardo <jdlutzardo@izertis.com>
Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Signed-off-by: delehef <franklin.delehelle@odena.eu>
Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com>
Signed-off-by: garyschulte <garyschulte@gmail.com>
Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu>
Signed-off-by: Karim TAAM <karim.t2am@gmail.com>
Signed-off-by: matkt <karim.t2am@gmail.com>
Signed-off-by: Matthew Whitehead <matthew1001@gmail.com>
Signed-off-by: Matt Whitehead <matthew1001@hotmail.com>
Signed-off-by: Diego López León <dieguitoll@gmail.com>
Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net>
Co-authored-by: David Lutzardo <jdlutzardo@izertis.com>
Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net>
Co-authored-by: Simon Dudley <simon.dudley@consensys.net>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: delehef <franklin.delehelle@odena.eu>
Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com>
Co-authored-by: garyschulte <garyschulte@gmail.com>
Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com>
Co-authored-by: matkt <karim.t2am@gmail.com>
Co-authored-by: Matt Whitehead <matthew1001@gmail.com>
Co-authored-by: Diego López León <dieguitoll@gmail.com>
gfukushima added a commit to gfukushima/besu that referenced this pull request Dec 15, 2023
…er#6185)

* Add RockDB Subcommand for printing usage per column family
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

* changed output to follow a MD table notation.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
jflo pushed a commit to jflo/besu that referenced this pull request Dec 18, 2023
…er#6185)

* Add RockDB Subcommand for printing usage per column family
Signed-off-by: Simon Dudley <simon.dudley@consensys.net>

* changed output to follow a MD table notation.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

---------

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com>
Signed-off-by: jflo <justin+github@florentine.us>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
TeamGroot GH issues worked on by Groot Team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants