-
Notifications
You must be signed in to change notification settings - Fork 841
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Backward sync invalid block causes Out of Memory error #6138
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…bad block cache Signed-off-by: Jason Frame <jason.frame@consensys.net>
|
jframe
force-pushed
the
bws_badblock_oom
branch
from
November 7, 2023 06:37
00b4e16
to
0249f89
Compare
jframe
changed the title
Bws badblock oom
Backward sync invalid block causes Out of Memory error
Nov 8, 2023
macfarla
approved these changes
Nov 8, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM just a typo in changelog entry
Signed-off-by: Jason Frame <jason.frame@consensys.net>
jflo
approved these changes
Nov 8, 2023
siladu
approved these changes
Nov 8, 2023
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice find!
jflo
pushed a commit
to jflo/besu
that referenced
this pull request
Nov 10, 2023
Signed-off-by: Jason Frame <jason.frame@consensys.net> (cherry picked from commit 701cbb0)
Merged
jflo
added a commit
that referenced
this pull request
Nov 10, 2023
* Unsigned timestamps and blob gas used (#6046) * lots of places an unsigned timestamp is a problem * adds unchecked annotations to OptionalUnsignedLong rpc parameter type --------- Signed-off-by: Justin Florentine <justin+github@florentine.us> (cherry picked from commit a90ea05) * [MINOR] ux improvements (#6049) * log command line option that is affected * made plugins summary log part of config overview * check for null plugin context Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit e8bca61) * Update changelog release (#6062) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> (cherry picked from commit f810887) * Dencun corner cases (#6060) * cherry pick changes from https://github.com/hyperledger/besu/pull/6054/files\#diff-22b78733e37a697fa8d1d8a02d2a87fe5ccea9cf67c34ce5e6311f024c14abd6L643-R738 Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * cherry pick changes from https://github.com/hyperledger/besu/pull/6054/files\#diff-61db834b59eae5ce5c438462505de1add8fa244deda830742060d15f668a9806R39-R44 Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * formatting Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> * update the EIP-6110 acceptance tests Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit 9d9fe8c) * add retry logic on sync pipeline for rocksdb issue (#6004) * add retry logic for sync pipeline with rocksdb issue Signed-off-by: Karim TAAM <karim.t2am@gmail.com> (cherry picked from commit c839a3b) * Mining options refactor (#6027) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit 20a82d4) * Update reference tests to Cancun (#6054) * Update reference tests to Cancun Update reference tests to cancun tests. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * update the subrepo Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * update the EIP-6110 acceptance tests Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * update to develop Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> --------- Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit 909649f) * Optimize GetPooledTransactionsFromPeerTask with HashSet (#6071) Switch from using a List to a HashSet for transaction hashes in GetPooledTransactionsFromPeerTask to improve performance. Signed-off-by: Suyash Nayan <suyashnyn1@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit a2dbb82) * Decouple TrieLogManager and CachedWorldStorageManager (#6072) Separate out the concepts of world state caching and trie log management. Remove AbstractTrieLogManager and make TrieLogManager the concrete implementation. Signed-off-by: Simon Dudley <simon.dudley@consensys.net> (cherry picked from commit 20f518e) * Implement miner_setMinPriorityFee and miner_getMinPriorityFee (#6080) * Refactor mining options Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Fix null pointer exception Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * fix another null pointer exception Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * uncomment code Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Move miner options tests Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Unit test fixes Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Removed the commented code Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * WIP Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * WIP Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * New miner option: min-priority-fee Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Remove not relevant for this feature Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Remove not relevant for this feature Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Fix javadoc Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Remove code not belonging to this PR Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * coinbase is an updatable parameter Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Move MiningOptions to upper package Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Fix coinbase for *bft Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Implement methods to get and set min priority fee Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix spotless Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Apply suggestions from code review Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Update besu/src/main/java/org/hyperledger/besu/cli/BesuCommand.java [skip-ci] Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> * Add new config option to everything config Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Fix unit test Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> * Accept PR suggestions Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> --------- Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Signed-off-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> (cherry picked from commit 3310c41) * Use Bytes Trie to track warm addresses (#6069) * Use Bytes Trie to track warm addresses Move from a java HashSet to a custom Trie based on bytes to store the warm addresses, creates, and self-destructs. This avoids needing to calculate java hashes or engage in using custom Comparators. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * codeql scan Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> --------- Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit af011d2) * Migrate Operand Stack to Growing Stack (#6068) To improve the performance of CALL operations move the OperandStack and ReturnStack to growing stacks instead of a fully allocated stack. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit 949c724) * Remove a BonsaiWorldStateProvider constructor (#6090) Was only used for supporting test code and can instead reuse static factory from InMemoryKeyValueStorageProvider Signed-off-by: Simon Dudley <simon.dudley@consensys.net> (cherry picked from commit 9a22703) * add RpcEndpointService to thread runner (#6091) Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit 7ac8af0) * ETC 'Spiral' network upgrade (#6078) * Add 23.10.2 section to changelog Signed-off-by: Diego López León <dieguitoll@gmail.com> * Set ENR tree for DNS discovery for Mordor network Signed-off-by: Diego López León <dieguitoll@gmail.com> * Add ECIP-1109: 'Spiral' network upgrade support Signed-off-by: Diego López León <dieguitoll@gmail.com> --------- Signed-off-by: Diego López León <dieguitoll@gmail.com> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit f58f6cf) * Trigger contextEnter/Exit for all frames, including root (#6074) * Trigger contextEnter/Exit for all frames, including root * Differentiate between context entry and re-entry in `OperationTracer` * Update evm/src/test/java/org/hyperledger/besu/evm/processor/AbstractMessageProcessorTest.java Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Signed-off-by: delehef <github@odena.eu> --------- Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> Signed-off-by: delehef <github@odena.eu> Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit 0345b24) * feat: add a way to read memory without altering the word capacity (#6073) * feat: add a way to read memory without altering the word capacity * add tests Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> * Fix read-past-end * Do not abuse method overload * Update CHANGELOG.md Signed-off-by: delehef <github@odena.eu> Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> * add tests for MessageFrame.shadowReadMemory Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> * Straddled reads tests Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> --------- Signed-off-by: Franklin Delehelle <franklin.delehelle@odena.eu> Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> Signed-off-by: delehef <github@odena.eu> Signed-off-by: delehef <franklin.delehelle@odena.eu> Co-authored-by: Daniel Lehrner <daniel.lehrner@consensys.net> (cherry picked from commit edf23cb) * renamed env field (#6096) Signed-off-by: Justin Florentine <justin+github@florentine.us> (cherry picked from commit 67ef9e0) * [MINOR] Upgrade netty and grpc (#6100) * Upgrade netty and grpc * fix verification file Signed-off-by: Simon Dudley <simon.dudley@consensys.net> --------- Signed-off-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit accac1c) * TraceService: return results for transactions in block (#6087) * TraceService: return results for transactions in block Signed-off-by: Daniel Lehrner <daniel.lehrner@consensys.net> (cherry picked from commit a60b31b) * RPC Parameters to accept input and data field (#6094) * Accept input or data as payload for RPC calls Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add json new rpc valid and invalid request to test the changes Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Change JsonCallParameter signature to avoid duplicating constructor Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> * Add changelog Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> --------- Signed-off-by: Gabriel Fukushima <gabrielfukushima@gmail.com> (cherry picked from commit 6dd558b) * Fix some typos (#6093) * Fix some typos * Update plugin version Signed-off-by: GoodDaisy <90915921+GoodDaisy@users.noreply.github.com> --------- Signed-off-by: GoodDaisy <90915921+GoodDaisy@users.noreply.github.com> (cherry picked from commit e814886) * Correct reference test blobgas calculation (#6107) * Correct reference test blobgas calculation Fix tpyo that resulted in an NPE in t8n blob gas calculations. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * changelog Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> * spotless Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> --------- Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit d0a6a70) * Restore javadoc and sources jar as trusted artifacts (#6109) Makes Idea happy again as documented here https://docs.gradle.org/6.8.3/userguide/dependency_verification.html#sec:skipping-javadocs Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> (cherry picked from commit 311570f) * Add API to set and get the minGasPrice at runtime (#6097) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> (cherry picked from commit de8ca10) * Don't put NONCE_TOO_LOW transactions into the invalid nonce cache (#6067) * Don't put NONCE_TOO_LOW transactions into the invalid nonce cache Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update unit tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Use list of errors to ignore Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> --------- Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> Signed-off-by: Matt Whitehead <matthew1001@gmail.com> Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> (cherry picked from commit 84dee29) * Journaled world state (#6023) Introduce a new Journaled World State Updater. Within a transaction it keeps one copy of account and storage state, restoring previous revisions on reverts and exceptional halts. This updater only supports post-merge semantics with regard to empty accounts, namely that they do not exist in world state. Adds an EvmConfiguration option for stacked vs journaled updater, and wire it in where needed. The staked updater is default mode, which is the current behavior prior to this patch. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit 094c841) * setMinPriorityFee - Return Invalid Param when invalid and use hexadecimal instead of Long (#6099) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> (cherry picked from commit 7acdd87) * [MINOR] - Add 23.10.1 correct changelog (#6110) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> (cherry picked from commit 236779d) * Add transaction selector based on min priority fee parameter (#6083) Signed-off-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> (cherry picked from commit 41b9575) * Add config option to clique to allow not creating empty blocks (#6082) Signed-off-by: Jason Frame <jason.frame@consensys.net> (cherry picked from commit 6a2d41f) * load all accounts into cache (#6101) There was a slight problem on the bonsai side because all account reads did not go through a single method. One of the two add the account to the cache but the other did not. This had two consequences: Less good performance because certain accounts had to be read several times and also all account reads were not marked in the trielog. This will fix both problems. Signed-off-by: Karim TAAM <karim.t2am@gmail.com> (cherry picked from commit d049cb3) * Reference Tests v13 (#6114) v13 of the official Ethereum Reference Tests. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit cea3d8a) * clean up the ProcessableBlockHeader (#6117) * clean up Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> (cherry picked from commit 646c5a3) * Log missing chain head as warning, not trace (#6127) Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> (cherry picked from commit 83ae69a) * Update RPCs for yParity (#6119) Update the GraphQL and JSON-RPC endpoints to provide `yParity` instead of `v` for non-legacy transactions. Update the JSON-RPC tests to use the Hive data. Add tests for Shanghai and Cancun Blocks. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit 2284242) * Make tracer in block building block aware (#6135) Signed-off-by: stefan.pingel@consensys.net <stefan.pingel@consensys.net> (cherry picked from commit f42f8c1) * Revert "Discard invalid transaction" warning logs (#6137) * Revert "Log missing chain head as warning, not trace (#6127)" * leave chain head at warning, just revert the discard warning Signed-off-by: garyschulte <garyschulte@gmail.com> (cherry picked from commit 914ab79) * Reverse sort order (#6106) * Reverse added order and sequence number Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Remove extraneous 'addedAt' check Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> --------- Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> (cherry picked from commit 0203092) * Force tx replacement price bump to zero when zero base fee market is configured (#6079) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> (cherry picked from commit 636ad8a) * Time limited block creation (#6044) Signed-off-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> (cherry picked from commit 8319fae) * Limit memory used in handling invalid blocks (#6138) Signed-off-by: Jason Frame <jason.frame@consensys.net> (cherry picked from commit 701cbb0) * Delete leftPad when capturing the stack before and after a frame execution (#6102) * Delete leftPad when capturing the stack before and after the execution * Still use leftPad when displaying the stack in the output (ex. for debug_traceTransaction) * Fix integration test * Use StringBuilder to left pad the hex representation of a 32 bytes Signed-off-by: Ameziane H <ameziane.hamlat@consensys.net> (cherry picked from commit 9710a9a) * fix yParity flakey test (#6151) Fix the flakeiness in EthGetTransactionByHashTest as well as some other sonar identified cleanup. Signed-off-by: Danno Ferrin <danno.ferrin@swirldslabs.com> (cherry picked from commit 4b58d07) * Apply the same reverse sort order as https://github.com/hyperledger/b… (#6146) * Apply the same reverse sort order as #6106 but to the base fee sorter Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Fix unit tests Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update eviction unit tests to expect highest-sequence TXs be evicted first Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Update change log Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> * Spotless fixes Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> --------- Signed-off-by: Matthew Whitehead <matthew1001@gmail.com> (cherry picked from commit 8afad41) --------- Co-authored-by: Sally MacFarlane <macfarla.github@gmail.com> Co-authored-by: Gabriel-Trintinalia <gabriel.trintinalia@consensys.net> Co-authored-by: matkt <karim.t2am@gmail.com> Co-authored-by: Fabio Di Fabio <fabio.difabio@consensys.net> Co-authored-by: Danno Ferrin <danno.ferrin@swirldslabs.com> Co-authored-by: Suyash Nayan <89125422+7suyash7@users.noreply.github.com> Co-authored-by: Simon Dudley <simon.dudley@consensys.net> Co-authored-by: Diego López León <dieguitoll@gmail.com> Co-authored-by: delehef <franklin.delehelle@odena.eu> Co-authored-by: daniellehrner <daniel.lehrner@consensys.net> Co-authored-by: Gabriel Fukushima <gabrielfukushima@gmail.com> Co-authored-by: GoodDaisy <90915921+GoodDaisy@users.noreply.github.com> Co-authored-by: Matt Whitehead <matthew1001@gmail.com> Co-authored-by: Jason Frame <jason.frame@consensys.net> Co-authored-by: Stefan Pingel <16143240+pinges@users.noreply.github.com> Co-authored-by: garyschulte <garyschulte@gmail.com> Co-authored-by: ahamlat <ameziane.hamlat@consensys.net>
Closed
NickSneo
pushed a commit
to NickSneo/besu
that referenced
this pull request
Nov 12, 2023
Signed-off-by: Jason Frame <jason.frame@consensys.net>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
PR description
Besu fails to import a block due to an invalid block error during BWS and then fails with an OutOfMemoryError.
The fix is to limit the included blocks and block headers in the
onBadChain
event. I have set the limit to be the same as what is set for the maximum number of entries cached in theBadBlockManager
. TheBadBlockManager
is currently the only listener for this event and the limit of 100 also seems reasonable for theonBadChain
event.Heap dump analysis
Fixed Issue(s)
fixes #6118