Add --X-trie-log subcommand #6303

gfukushima · 2023-12-15T04:10:01Z

PR description

This is a more direct approach to the previous attempt in #6188.
Tests showed that the previous approach was taking too long (3 hours) to prune a 3.5 months old (keeping the lastest 512 only).
The approach this PR takes is faster since it resets the column family and only retain the necessary trie logs (saved in files in batches of 20_000 trie logs).
Drawback of this approach is that we do store the trie logs in disk (in files) before dropping the column. Realistic speaking there's little to no reason to actually keep many more trie logs than the default(512).
In any case, I've made some adjustments to the code to allow this to go as far as a user might like.

Ran tests on nodes retaining 512 / 20_000 / 250_000 / 1M trie logs, all instances had approx 1.8M trie logs in the db when node was stopped for prunning.

Below some of the logs condensed

Real use case scenario - 512 trie logs retained (1 second)

2023-12-22 05:40:22.409+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 1)...
2023-12-22 05:40:22.416+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2023-12-22 05:40:22.692+00:00 | main | INFO  | TrieLogHelper | Clear trie logs...
2023-12-22 05:40:23.131+00:00 | main | INFO  | TrieLogHelper | Restoring trie logs retained from batch 1...
2023-12-22 05:40:23.446+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2023-12-22 05:40:23.452+00:00 | main | INFO  | TrieLogHelper | Prune ran successfully. Enjoy some disk space back! 🚀

a more extreme case where for some reason someone has been running an archieve and decides to keep only the latest 1M trie logs - 1M trie logs retained (approx 2 hours)

2024-01-03 01:48:18.042+00:00 | main | INFO  | BonsaiWorldStateKeyValueStorage | Bonsai flat db mode found PARTIAL
2024-01-03 01:48:18.806+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 1)...
2024-01-03 01:48:18.809+00:00 | main | INFO  | TrieLogHelper | Obtaining trielogs from db, this may take a few minutes...
2024-01-03 01:48:34.970+00:00 | main | INFO  | TrieLogHelper | Saving trie logs to retain in file (batch 2)...

2024-01-03 03:42:11.016+00:00 | main | INFO  | TrieLogHelper | Deleting files...
2024-01-03 03:42:11.029+00:00 | main | INFO  | TrieLogHelper | Prune ran successfully. Enjoy some disk space back! 🚀

Fixed Issue(s)

Part of #5390

github-actions · 2023-12-15T04:10:17Z

I thought about documentation and added the doc-change-required label to this PR if updates are required.
I thought about the changelog and included a changelog update if required.
If my PR includes database changes (e.g. KeyValueSegmentIdentifier) I have thought about compatibility and performed forwards and backwards compatibility tests

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java

jframe · 2023-12-15T05:26:30Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+
+  private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {
+
+    File file = new File(trieLogFile);


Wold be better to use File.createTempFile so it's created in the temp directory and we won't have any write permissions issues

We need the file to exist after we finish the run if for some reason we fail to reload the trielogs into the db

jframe · 2023-12-15T05:27:12Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+  }
+
+  private static void deleteTrieLogFile() {
+    File file = new File(trieLogFile);


When we create the file we could use the File.deleteOnExit so we don't need to worry about manually deleting this file

jframe · 2023-12-15T05:29:36Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+                .ifPresent(trieLog -> trieLogsToRetain.put(hash.toArrayUnsafe(), trieLog));
+          });
+      out.println("Saving trielogs to retain in file...");
+      saveTrieLogsInFile(trieLogsToRetain);


Not sure how the fail-safe is being used. Expected to see that if we got an error clearing and inserting entries we ready from the file or allow specifying an existing backup file to restore from.

Added a comment and instructions that will be logged for the user. Basically the idea here is, we retain the logs in a file in case something goes wrong.
In that case the user can re-run the subcommand and we will read the trielogs from the file since the column family might have been dropped already.

jframe · 2023-12-15T05:46:57Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+      saveTrieLogsInFile(trieLogsToRetain);
+    } else {
+      // try to read the triLogs from the flatfile
+      trieLogsToRetain = readTrieLogsFromFile();


Don't understand how reading from the file helps in this case. We didn't find all the hashes but also haven't created the file at this point either unless we are relying running the command again to read the previous state

I think this is a failsafe in for a subsequent execution where something went terribly wrong during the truncation and restore-from-file process.

what would be nice, but not idiomatic, is if we had a few "are you sure?" or "do you want to?" prompts that could be bypassed by -y similar to ubuntu apt commands

jframe · 2023-12-15T05:52:43Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+        (key, value) -> {
+          updater.getTrieLogStorageTransaction().put(key, value);
+        });
+    updater.getTrieLogStorageTransaction().commit();


If this fails then do have a way of restoring the trielog from the backup?

Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

…tIdentifier.reset() Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

…emory Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

garyschulte · 2023-12-15T19:01:17Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/RocksDbUsageHelper.java

@@ -34,17 +34,17 @@ static void printUsageForColumnFamily(
      final RocksDB rocksdb, final ColumnFamilyHandle cfHandle, final PrintWriter out)
      throws RocksDBException, NumberFormatException {
    final String size = rocksdb.getProperty(cfHandle, "rocksdb.estimate-live-data-size");
+    final String numberOfKeys = rocksdb.getProperty(cfHandle, "rocksdb.estimate-num-keys");


changes here seem fine but unrelated. Every time I look at this class name though I think it is a CLI Usage helper, like the output that gets printed when one asks for help or gets command usage wrong. completely unrelated, but if we are going to make changes in this pr ... :)

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

garyschulte · 2023-12-15T19:13:20Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+  private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {
+
+    File file = new File(trieLogFile);
+    IdentityHashMap<byte[], byte[]> trieLogs = new IdentityHashMap<>();


trielogs can vary in size, and the retention count is user configurable. We could easily blow up the heap with bad settings. We might want to add some sanity check on the file size here, and perhaps also in trielog streaming/save to file.

Agree, users could eventually set the layersToRetain to a high number and this could blow up. Is there real scenario where we should handle a high number having in mind that rolling back too many layer is not something feasible in bonsai (afaik)?

Trielogs can be the basis of a specific worldstate for debug_ and trace_ endpoints. But really if that were the case I imagine the node would be an archive node. I think streaming to and from disk would likely just be more bulletproof.

Not blocking feedback, just seems a robust mechanism

I've changed this so we process it in batches now so it should be a safer mechanism.

garyschulte

Looks good. I think it wouldn't be too difficult to stream trielogs to and from the file directly to prevent OOM and heap issues.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

matkt · 2023-12-20T08:48:37Z

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

+      LOG.info("Restoring trie logs retained from batch {}...", batchNumber);
+      recreateTrieLogs(rootWorldStateStorage, batchNumber, batchFileNameBase);
+    } catch (IOException | ClassNotFoundException e) {
+      LOG.error("Error recreating trie logs from batch {}: {}", batchNumber, e.getMessage());


What should we do at this point? Do we have a way to restart from an existing restore file?

I'm going to change this to throw a new Runtime exception so we abort the process here, meaning if you re-run the command it's gonna detect the files exist and skip the save to file part but it will still reset the db and try to restore from file.

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

...reum/core/src/main/java/org/hyperledger/besu/ethereum/trie/bonsai/trielog/TrieLogPruner.java

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

…trie-log-subcommand-2

* mark deleted slot during clear storage step (hyperledger#6305) Signed-off-by: Karim Taam <karim.t2am@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> * made directory structure of tests match source; fixed one typo (hyperledger#6337) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * migrate controller tests to junit 5 (hyperledger#6338) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * add new forkids for testnets, update forkid test to Junit5, no longer need named network specific trusted setups (hyperledger#6322) Signed-off-by: jflo <justin+github@florentine.us> * Fix trielog shipping issue during self destruct (hyperledger#6340) * fix trielog shipping issue Signed-off-by: Karim Taam <karim.t2am@gmail.com> * bump gradle properties version and adjust changelog to match release (hyperledger#6347) Signed-off-by: garyschulte <garyschulte@gmail.com> * finalized cancun spec (hyperledger#6351) * finalized cancun spec Signed-off-by: jflo <justin+github@florentine.us> * finalized cancun spec Signed-off-by: jflo <justin+github@florentine.us> --------- Signed-off-by: jflo <justin+github@florentine.us> * Optimize RocksDB WAL file (hyperledger#6328) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Make RPC reason settable, pass execution failure reason in RPC error message (hyperledger#6343) * Make RPC reason settable, pass execution failure reason in RPC error message Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update unit tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update change log Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update integration tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> --------- Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * TestWatcher junit5 (hyperledger#6339) * TestWatcher junit5 * add test class and method name to context * moved the testwatcher junit5 function to a new junit5 superclass * one qbft test to junit5 superclass Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> * Migrate BFT tests to junit 5 (hyperledger#6350) * bft tests to junit 5 * base class for pki extend AcceptanceTestBaseJunit5 * try/catch in case of empty optionals * fixed parameterization method Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * fixing on selfdestruct (hyperledger#6359) Signed-off-by: Karim Taam <karim.t2am@gmail.com> * migrate clique tests fully to junit5 (hyperledger#6362) * migrate clique tests fully to junit5 Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * fixed link to logging docs (hyperledger#6366) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Move logging to RunnerBuilder (hyperledger#6367) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Use synchronized call to access the chain head block in `eth_estimateGas` (hyperledger#6345) * Use synchronized call to access the chain head block in estimateGas() Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Add error log entries when throwing internal error from estimateGas() Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update unit tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update changelog Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> --------- Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Add --X-trie-log subcommand (hyperledger#6303) * Add x-trie-log subcommand for one-off trie log backlog prune Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> --------- Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> * fix typos (hyperledger#6368) Signed-off-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> * Added alias --sync-min-peers for --fast-sync-min-peers (hyperledger#6372) * sync-min-peers as an alias * added unit tests Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Fix: Fallback to getName when canonicalName is null in BlockHeaderValidator DEBUG log (hyperledger#6332) * fallback to simple name when canonical name is null * use getName instead of getSimpleName to include the package name Signed-off-by: Manoj P R <manojpramesh@gmail.com> --------- Signed-off-by: Manoj P R <manojpramesh@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * fix: use UID 1000 for besu user (hyperledger#6358) (hyperledger#6360) The openjdk-latest Docker image is using UID 1001 for besu, because its base image ubuntu:23.10 now contains a default "ubuntu" user with UID 1000. (This UID change causes the besu user with UID 1001 to not have access to files created for past versions with UID 1000.) We now remove the default ubuntu user and explicitly use UID 1000 when creating the besu user. Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> * Ignore generated files when running the spdx license check (hyperledger#6379) Signed-off-by: Meredith Baxter <mbaxter.dev@gmail.com> * full sync - don't fail startup if sync-min-peers specified (hyperledger#6373) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * Copy also computed fields, when doing a Transaction detached copy (hyperledger#6329) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Disable txpool when not in sync (hyperledger#6302) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump to nex release snapshot 24.1.1 (hyperledger#6383) * release next snapshot 24.1.1 Signed-off-by: garyschulte <garyschulte@gmail.com> * move recent changelog items to 24.1.1-SNAPSHOT Signed-off-by: garyschulte <garyschulte@gmail.com> --------- Signed-off-by: garyschulte <garyschulte@gmail.com> * Correct Tangerine Whistle definition in Fluent EVM APIs. (hyperledger#6382) The fluent API incorrectly added the code size limit in Tangerine Whistle instead of first adding it in Spurious Dragon. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * [MINOR] Fix pki tests condition check on mac (hyperledger#6387) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Upgrade dependencies (hyperledger#6377) * Bump com.github.oshi:oshi-core to 6.4.10 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.github.tomakehurst to org.wiremock 3.3.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.google.auto.service:auto-service to 1.1.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.google.dagger group to 2.50 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.graphql-java:graphql-java to 21.3 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.splunk.logging:splunk-library-javalogging to 1.11.8 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump com.squareup.okhttp3:okhttp to 4.12.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump commons-io:commons-io to 2.15.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump dnsjava:dnsjava to 3.5.3 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump info.picocli group to 4.7.5 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump io.grpc group to 1.60.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump io.kubernetes:client-java to 18.0.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump io.netty group to 4.1.104.Final Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump net.java.dev.jna:jna to 5.14.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.apache.commons:commons-compress to 1.25.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.apache.commons:commons-lang3 to 3.14.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.apache.commons:commons-text to 1.11.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.apache.logging.log4j group to 2.22.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Redorder io.tmio group Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.assertj:assertj-core to 3.25.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.bouncycastle group to 1.77 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.fusesource.jansi:jansi to 2.4.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.immutables group 2.10.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.java-websocket:Java-WebSocket to 1.5.5 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.jetbrains.kotlin:kotlin-stdlib to 1.9.22 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.junit.jupiter group to 5.10.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.jupnp group to 2.7.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.rocksdb:rocksdbjni to 8.9.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.slf4j group to 2.0.10 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.springframework.security:spring-security-crypto to 6.2.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.testcontainers:testcontainers to 1.19.3 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.web3j group to 4.10.3 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Bump org.xerial.snappy:snappy-java to 1.1.10.5 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Regenerate gradle verification metadata Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update commons-codec:commons-codec to 1.16.0 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update org.junit.vintage:junit-vintage-engine to 5.10.1 Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update CHANGELOG Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> --------- Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * add a fallback for docker detection on Mac (hyperledger#6356) Signed-off-by: garyschulte <garyschulte@gmail.com> * Fix test flackyness of acceptanceTestsPermissioning (hyperledger#6384) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Upgrade `com.fasterxml.jackson` dependencies (hyperledger#6378) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Use mining beneficiary from protocol spec in TraceServiceImpl (hyperledger#6390) * use mining beneficiary from protocol spec Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> --------- Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> * Update verification metadata and allowed licenses for Linea-Besu --------- Signed-off-by: Karim Taam <karim.t2am@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: jflo <justin+github@florentine.us> Signed-off-by: garyschulte <garyschulte@gmail.com> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Signed-off-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> Signed-off-by: Manoj P R <manojpramesh@gmail.com> Signed-off-by: Hal Blackburn <hwtb2@cam.ac.uk> Signed-off-by: Meredith Baxter <mbaxter.dev@gmail.com> Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Co-authored-by: Karim TAAM <karim.t2am@gmail.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Justin Florentine <justin+github@florentine.us> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Matt Whitehead <matthew1001@gmail.com> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: vuittont60 <81072379+vuittont60@users.noreply.github.com> Co-authored-by: Manoj P R <manojpramesh@gmail.com> Co-authored-by: Hal Blackburn <hal.blackburn@gmail.com> Co-authored-by: mbaxter <mbaxter.dev@gmail.com> Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: daniellehrner <daniel.lehrner@consensys.net>

gfukushima changed the title ~~X trie log subcommand~~ Add --X-trie-log subcommand Dec 15, 2023

gfukushima force-pushed the x-trie-log-subcommand-2 branch 2 times, most recently from 20960ce to 9f149a5 Compare December 15, 2023 05:53

jframe reviewed Dec 15, 2023

View reviewed changes

siladu and others added 8 commits December 15, 2023 16:00

Add x-trie-log subcommand for one-off backlog prune

16c0a49

Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

long -> int

7dd4928

Signed-off-by: Simon Dudley <simon.dudley@consensys.net> Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Removed banned method

bf2b098

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Preload process stream in parallel

e67ae51

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Drop unwanted trielogs and keep reatain layers only

9b4e0c9

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Add output to user and cleanup refactor

0b9fe83

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

small tweak to display cf that had reference dropped by RocksDbSegmen…

426848e

…tIdentifier.reset() Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

spotless

7401b59

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

gfukushima force-pushed the x-trie-log-subcommand-2 branch from 9f149a5 to 7401b59 Compare December 15, 2023 06:00

gfukushima added 4 commits December 15, 2023 16:27

Fix classes that changed package

1b7fb72

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

spotless

11e6b05

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Code review

f2d01e2

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Only clear DB when we have the exact amount of trie logs we want in m…

04f1aaa

…emory Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

non-fungible-nelson requested a review from matkt December 15, 2023 14:37

garyschulte reviewed Dec 15, 2023

View reviewed changes

besu/src/main/java/org/hyperledger/besu/cli/subcommands/storage/TrieLogHelper.java Outdated Show resolved Hide resolved

garyschulte reviewed Dec 15, 2023

View reviewed changes

gfukushima added 3 commits December 18, 2023 13:02

Trielogs stream to and from file to avoid possibly OOM

2f01c5a

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Process trie logs in chunks to avoid OOM

56e4c8e

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

save and read in batches to handle edge cases

78561b0

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

gfukushima added the TeamGroot GH issues worked on by Groot Team label Dec 19, 2023

gfukushima added 2 commits December 20, 2023 14:42

save and read files to/from database dir

42c72cf

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Merge branch 'main' into x-trie-log-subcommand-2

9961fc2

gfukushima marked this pull request as ready for review December 20, 2023 06:31

gfukushima requested a review from garyschulte December 20, 2023 06:31

matkt reviewed Dec 20, 2023

View reviewed changes

gfukushima added 3 commits December 21, 2023 15:20

add unit tests and PR review fixes

9389540

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Merge branch 'main' into x-trie-log-subcommand-2

e3d4fbc

spdx

c7144fe

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

gfukushima mentioned this pull request Dec 21, 2023

Removal of old trie logs specified by historical block limit #5390

Closed

21 tasks

gfukushima added 2 commits December 21, 2023 20:52

Fix unit tests directory creation and deletion

20b0ba5

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Merge branch 'main' into x-trie-log-subcommand-2

e7d175c

jframe reviewed Jan 4, 2024

View reviewed changes

gfukushima added 3 commits January 4, 2024 16:20

PR review

b214bf2

Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com>

Merge branch 'main' into x-trie-log-subcommand-2

aa75348

Merge remote-tracking branch 'origin/x-trie-log-subcommand-2' into x-…

2bc0732

…trie-log-subcommand-2

gfukushima mentioned this pull request Jan 8, 2024

Import export trie log #6363

Merged

Merge branch 'main' into x-trie-log-subcommand-2

deec021

jframe approved these changes Jan 8, 2024

View reviewed changes

Merge branch 'main' into x-trie-log-subcommand-2

5d7b68e

gfukushima merged commit e51e042 into hyperledger:main Jan 8, 2024
18 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add --X-trie-log subcommand #6303

Add --X-trie-log subcommand #6303

gfukushima commented Dec 15, 2023 •

edited

Loading

github-actions bot commented Dec 15, 2023

jframe Dec 15, 2023

gfukushima Dec 15, 2023

jframe Dec 15, 2023

jframe Dec 15, 2023

gfukushima Dec 15, 2023

jframe Dec 15, 2023

garyschulte Dec 15, 2023

jframe Dec 15, 2023

garyschulte Dec 15, 2023

garyschulte Dec 15, 2023

gfukushima Dec 17, 2023 •

edited

Loading

garyschulte Dec 17, 2023

gfukushima Jan 3, 2024

garyschulte left a comment

matkt Dec 20, 2023

gfukushima Dec 20, 2023


		private static IdentityHashMap<byte[], byte[]> readTrieLogsFromFile() throws IOException {

		File file = new File(trieLogFile);

Add --X-trie-log subcommand #6303

Add --X-trie-log subcommand #6303

Conversation

gfukushima commented Dec 15, 2023 • edited Loading

PR description

Fixed Issue(s)

github-actions bot commented Dec 15, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gfukushima Dec 17, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

garyschulte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gfukushima commented Dec 15, 2023 •

edited

Loading

gfukushima Dec 17, 2023 •

edited

Loading