This repository has been archived by the owner on Feb 1, 2024. It is now read-only.
Add Blocks at a Configured Depth to Pruning Queue #1689
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
peterschwarz
requested review from
vaporos,
boydjohnson,
jsmitchell,
agunde406,
aludvik and
nick-drozd
aludvik
approved these changes
Jun 4, 2018
Remove #[macro_use] for cpython, as it is no longer necessary. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
from
df17664 to
9e4c206
Compare
nick-drozd
reviewed
Jun 4, 2018
validator/src/journal/chain.rs
Outdated
| state.state_prune_manager.update_queue( | ||
| &result | ||
| .new_chain | ||
| .iter() |
There was a problem hiding this comment.
Would it make more sense to have the function take care of this .iter.map.collect stuff rather than make the caller do it?
There was a problem hiding this comment.
The state pruning manager operates on state root hashes, not blocks
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
from
9e4c206 to
c6e533c
Compare
nick-drozd
approved these changes
Jun 4, 2018
|
Oh, commit |
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
2 times, most recently
from
ee433d8 to
f6d005b
Compare
aludvik
approved these changes
Jun 5, 2018
Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
- Add validation for integer CLI args (they must be positive, greater than zero) - Parse the args as u32 before sending them to the config object Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
from
b3f6357 to
271c6dc
Compare
Adds the CLI argument and configuration option for state_pruning_block_depth. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Pass the state_pruning_block_depth configuration option through to the ChainController via the FFI interface. Signed-off-by: Peter Schwarz <peterx.schwarz@intel.com>
The StatePruningManager is responsible fore managing a queue of state roots and their respective block heights. It keeps track of unique hashes and supplies an execute method which will delete any hash below a certain depth. If there are state roots that compute the same hash value, they will be removed so as not to cause lost state roots for newer blocks. Any hash unsuccessfully pruned will be put back on the queue until such time as it can be removed. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
from
271c6dc to
cd76cf8
Compare
|
Added some new commits to handle the issue where an identical leaf is produced, and then is pruned from a prior state root hash. This approach is the most direct - scanning the change logs. Future changes could make this more efficient by managing this with an in-memory reference count. |
nick-drozd
approved these changes
Jun 7, 2018
When a block is committed to the block store, the added state root hashes and any uncommitted blocks' state root hashes are passed to the StatePruningManager. Also, if the block is of height greater than the state pruning depth, a block at that depth is added to the queue and the pruning operation is executed for that height. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Supply the global state database instance across the FFI boundary in order to create the a StatePruningManager Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Provide the state database to the python chain controller, in order to pass it through to the FFI rust layer. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Update tests to include a state database instance Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Add `next` to LmdbDatabaseReaderCursor, which previously only had `first` and `last`. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
Remove duplicate keys from the change log during pruning, to avoid removing leaf or intermediate nodes that may be referenced in future tries. Signed-off-by: Peter Schwarz <pschwarz@bitwise.io>
peterschwarz
force-pushed
the
STL-1220.pruning_queue
branch
from
cd76cf8 to
de0babb
Compare
aludvik
approved these changes
Jun 7, 2018
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR creates a pruning manager, which prunes state roots from a queue at predictable times. It is evaluated at chain commit time. 10 percent of the queue is pruned, which allows state roots to be returned to validity, if need be before they are finally pruned.
This PR does not solve the problem of abandoned forks, as they cannot be properly re-evaluated if a validator switches back to them. To properly cleanup state after a chain has been dropped, we will need the Block Tree Manager (See RFC 5). When uncommitted blocks expire from the Block Tree Manager, their state roots can be pruned.