Use linked-list to store versions #9
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This PR has been open for about two years, does not seem like there is interest to include it, Will keep the branch, but will close this PR. |
|
Yeah I think the benefits are kind of a matter of preference 🤷♂️ |
|
@cloudhead if you are interested in helping tendermint/tendermint#3769 was just opened about state-sync with a link to your proposal, your comments would be valuable to the design process |
tac0turtle
pushed a commit
that referenced
this pull request
Apr 9, 2022
* Add unbounded key string to KeyFormat * Add test vectors for unbounded length keys * Add some notes * update .gitignore * Add FastNode struct * WIP: make Get work with new FastNode * when retrieving fastnode fails, return errors vs. panic * add comments clarifying what index represents * make the linter happy * Add small tweaks to fast_node.go * add TODO & small linter tweaks * Update comment * update fast node cache in set * add debugging output when falling back to original logic * return error instead of panic * WIP: refactor set ops to work with fast store * update Set of mutable tree, begin unit testing * update GetVersioned to check fast nodes before trying the immutable * check fast node version before nil value check in get of immutable tree * fix small bugs and typos, continue writing unit tests for Set * unit test saveFastNodeVersion * simplify storing unsaved fast nodes * resolve a bug with not writing prefix for fast node to disk * remove fast nodes from disk on save and clear fast cache when version is deleted, fix all tests but random and with index * resolve an issue with randomized tests caused by the fast node cache not being cleared when latest version is saved * split unsaved fast node changes into additions and removals * save fast node removals * move fast node cache clearing to nodedb * use regular logic only when fast node version is greater than immutable tree' * clean up tree_random_test.go * clear unsaved fast node removals on rollback * fix randomized test failures caused by a typo in ndb DeleteVersion for loop * implement GetFast method to preserve Get with correct index return, convert unit tests from Get to GetFast where applicable * ensure Get and GetFast return the same values in tree_random_test.go * test fast node cache is live in random tree tests * improve mutable tree unit tests related to new functionality * clean up tree_test.go * implement GetVersionedFast to preserve the index in GetVersioned * restore accidentally deleted test in mutable tree test * spurious whitespace * refactor mutable tree * fix comment in mutable tree * add benchmark results * avoid redundant full tree search in GetFast of immutable tree when fast node is nil and tree is latest * fix naming for bench test get on random keys * use method for get latestversio in get fast * optimize GetFast, perform a refactor to ensure that fast nodes on disk are matched and better test * add latest bench * Fast Node Iteration (#7) * propagate errors from nodedb and avoid panicking * begin implementing fast node iteration * resolve rebase issue in random tests * fix iteration to deserialize fast node for extracting the correct value * finalzie iteration and let all unit tests pass * add benchmarks * merge GetVersioned and GetVersionedFast * remove fast node deletion from DeleteVersion and DeleteVersionRange and benchmark * fix and unit test iteration on mutable and immutable trees * implement tests for iterator and iterate, begin testing fast iterator * fix and unit test fast iterator * refactor iterate methods of mutable and immutable trees * resolve some warnings * remove old bench results * refactor bench tests for iteration * Fast Cache Migration (#9) * implement nodedb changes to set and get chain version from the database * implement and unit test upgrade to fast storage in mutable tree * refactor for auto upgrade to fast version in mutable tree contructor, load version and lazy load version * use proper functionality for getting latest version * remove unused error * fix comment * use fast version value in tests * spurious tab * fix style problems and remove redundant code in tests * Rename Get to GetWithIndex and GetFast to Get * refactor and clean up bench tests * remove extra byte from fast node length * clean up immutable tree * refactor nil tree or ndb error for the iterators and their tests * avoid exporting methods for getting unsaved additions and removals * refactor fast upgrade to read from tree instead of traversing disk nodes and orphans, update unit tests * remove unneeded comment * refer to storage version consistently across the project * fix more warnings * optimize removal of fast nodes from cache on deletion * small changes in teh mutable tree * correct storage version key * auto set fast version in SaveVersion * avoid exporting new methods of the immutable tree * go fmt * Fix comment in fast iterator Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * add extra comment for domain in fast iterator Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * add comments for moving the iterator before the first element * add comment for describing what mirror is in assertIterator of testutils * fix checking the mirror for descending iterator in tests * Update testutils_test.go with a comment Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * Update benchmarks/bench_test.go with a comment for runKnownQueriesFast Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * Update benchmarks/bench_test.go with a comment for runQueriesFast Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * export IsFastCacheEnabled and add an assert in bench tests * Update comment immutable_tree.go Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * Update comment for migration in mutable_tree.go Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> * simlify Iterate in mutable tree, add debug log for * Fast Cache - Downgrade - reupgrade protection and other improvements (#12) * add leaf hash to fast node and unit test * refactor get with index and get by index, fix migration in load version and lazy load version * use Get in GetVersioned of mutable tree * refactor non membership proof to use fast storage if available * bench non-membership proof * fix bench tests to work with the new changes * add downgrade-reupgrade protection and unit test * remove leaf hash from fast node * resolve multithreading bug related to iterators not being closed * clean up * use correct tree in bench tests * add cache to tree used to bench non membership proofs * add benc tests for GetWithIndex and GetByIndex * revert GetWithIndex and GetByIndex * remove unused import * unit test re-upgrade protection and fix small issues * remove redundant setStorageVersion method * fix bug with appending to live stage version to storage version and nit test * add comment for setFastStorageVersionToBatch * refactor and improve unit tests for reupgrade protection * rename ndb's isFastStorageEnabled to hasUpgradedToFastStorage and add comments * comment out new implementation for GetNonMembershipProof * update comments in nodedb to reflect the difference between hasUpgradedToFastStorage and shouldForceFastStorageUpdate * refactor nodedb tests * downgrade tendermint to 0.34.14 - osmosis's latest cosmos sdk does not support 0.35.0 * fix bug where fast storage was not enabled when version 0 was attempted to be loaded * implement unsaved fast iterator to be used in mutable tree (#16) * address comments from unsaved fast iterator PR * expose isUpgradeable method on mutable tree and unit test (#17) * expose isUpgradeable method on mutable tree and unit test * go fmt * resolve problems with rebasing * update CHANGELOG.md * tendermint v0.35.0 * fix String() bench * fix duplication lint in iterator_test.go * fix lint for tree.ndb.DeleteFastNode error return not checked * fix Error return value of `ndb.resetBatch` is not checked * fix Error return value of `ndb.traversePrefix` is not checked * fix Error: struct of size 64 bytes could be of size 56 bytes * fix Error: `comitted` is a misspelling of `committed` * fix issues in basic_test.go * address comments in fast_iterator.go * address comments in immutable tree * address comments in iterator.go * address comments in key_format.go * address remaining comments * fix Error: Error return value of `ndb.batch.Write` is not checked * fix Error: receiver name t should be consistent with previous receiver name tree for MutableTree * fix Error: struct of size 48 bytes could be of size 40 bytes * go fmt * more linter fixes * fix remaining linter problems * upgrade tm-db and comment out broken bencher databases * skip iterations for BenchmarkLevelDBLargeData bench * attempt to fix linter * force GC, no cache during migration, auto heap profile (#19) * force GC, no cache during migration, auto heap profile * resolve a potential deadlock from racing between reset and stop * fix small lint issue * remove logs and pprof logic * remove unused libraries * add comment explaining the reason for RAM optimizations * sync access to fast node cache to avoid concurrent write fatal error (#23) * update go.mod and go.sum to match master versions * revert #23 (sync access to fast node cache), fix bug related to old height export (#33) * Revert "sync access to fast node cache to avoid concurrent write fatal error (#23)" This reverts commit 2a1daf4. * return correct iterator in mutable tree * fix concurrent map panic when querying and comittting concurrently * avoid clearing fast node cache during pruning (#35) * fix data race related to VersionExists (#36) * fix data race related to VersionExists * use regular lock instead of RW in mutable_tree.go * hardcode fast node cache size to 100k * go fmt * restore proof_ics23.go * fix linter Co-authored-by: ValarDragon <dojha12@gmail.com> Co-authored-by: jtieri <37750742+jtieri@users.noreply.github.com> Co-authored-by: Dev Ojha <ValarDragon@users.noreply.github.com> Co-authored-by: Roman Akhtariev <34196718+akhtariev@users.noreply.github.com> Co-authored-by: Roman <34196718+r0mvn@users.noreply.github.com>
roysc
pushed a commit
to roysc/iavl
that referenced
this pull request
Jul 16, 2022
* implement nodedb changes to set and get chain version from the database * implement and unit test upgrade to fast storage in mutable tree * refactor for auto upgrade to fast version in mutable tree contructor, load version and lazy load version * use proper functionality for getting latest version * remove unused error * fix comment * use fast version value in tests * spurious tab * fix style problems and remove redundant code in tests
ulbqb
pushed a commit
to ulbqb/iavl
that referenced
this pull request
Jun 6, 2023
* ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Update comment Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add error checking for node hash * turn lengthByte into a const * Refactor linkNode * Update comment Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
ulbqb
pushed a commit
to ulbqb/iavl
that referenced
this pull request
Jun 6, 2023
* ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Add test to check add operation in deepsubtree: WIP * Add insert key functionality WIP * Try replicating non-inclusion proofs * Add sibling nodes to nonExistenceProofs * Refactor test code * Finish adding add functionality to dst * Add Remove functionality to dst * Fix linting errors * Fix GetSiblingNode * Remove more print statements * Add comment for each case in recursive set * Change which methods are exported and document exported methods * feat: Support Empty Hashes and Add constructor (cosmos#11) * Export siblings * Add deepsubtree constructor * Support empty root hashes * Use working hash instead of root.hash * Use tm-db instead of cosmos-db * Return nil in BuildTree * Address comments * Address more comments Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
ulbqb
pushed a commit
to ulbqb/iavl
that referenced
this pull request
Jun 12, 2023
* ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Update comment Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add error checking for node hash * turn lengthByte into a const * Refactor linkNode * Update comment Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
ulbqb
pushed a commit
to ulbqb/iavl
that referenced
this pull request
Jun 12, 2023
* ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Add test to check add operation in deepsubtree: WIP * Add insert key functionality WIP * Try replicating non-inclusion proofs * Add sibling nodes to nonExistenceProofs * Refactor test code * Finish adding add functionality to dst * Add Remove functionality to dst * Fix linting errors * Fix GetSiblingNode * Remove more print statements * Add comment for each case in recursive set * Change which methods are exported and document exported methods * feat: Support Empty Hashes and Add constructor (cosmos#11) * Export siblings * Add deepsubtree constructor * Support empty root hashes * Use working hash instead of root.hash * Use tm-db instead of cosmos-db * Return nil in BuildTree * Address comments * Address more comments Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com>
danwt
pushed a commit
to dymensionxyz/iavl
that referenced
this pull request
Apr 2, 2024
* feat: Deep tree structure with Updates (cosmos#13) * ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Update comment Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add error checking for node hash * turn lengthByte into a const * Refactor linkNode * Update comment Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * feat: Support Adds in a Deep Subtree (cosmos#8) * ICS23-style approach for DeepSubTree creation * Fix deepsubtree, all checks pass * Update documentation for deep subtrees * Fix deepsubtree to accomodate left/right paths both * Refactor test code * Refactor TestDeepSubtreeStepByStep * Address comments * Refactor code * Disable gocritic and unused * Add description * Refactor code to check err in Set * Modify traversal conditions to be clearer * feat: build deep subtree from ICS23 inclusion proofs (cosmos#9) * feat: build deep subtree from ICS23 inclusion proofs * feat: handle non-existence proofs * linter: goimports deepsubtree.go * refactor: addExistenceProofProof -> addExistenceProof * fix: un-hardcode size of byte array * Add more comments * Refactor ndb.Commit call outside for loop * verify that in the case that dst.root != nil that the root node hash matches the provided hash and check dst.root != nil first * Add strings and use strings.Repeat * Delete addPath and AddPath * Remove print statements * Refactor recomputeHash usage Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * return err directly Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Address linting checks * Use tm-db instead of cosmos-db * Fix linter * Add test to check add operation in deepsubtree: WIP * Add insert key functionality WIP * Try replicating non-inclusion proofs * Add sibling nodes to nonExistenceProofs * Refactor test code * Finish adding add functionality to dst * Add Remove functionality to dst * Fix linting errors * Fix GetSiblingNode * Remove more print statements * Add comment for each case in recursive set * Change which methods are exported and document exported methods * feat: Support Empty Hashes and Add constructor (cosmos#11) * Export siblings * Add deepsubtree constructor * Support empty root hashes * Use working hash instead of root.hash * Use tm-db instead of cosmos-db * Return nil in BuildTree * Address comments * Address more comments Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> * Add randomized tests for adding/removing keys (cosmos#16) * Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * Add tracing to Trees and Deep Subtrees (cosmos#18) * Add go fuzz tests * Add membership proof for existing keys * Build tree after adding membership proof * Make batch add fuzz tests work * Do not commit version twice for dst * Save version out of dst.Set condition * Set rootHash to workingHash * Fix edge cases * Refactor DST Non-Existence Proof * Change cacheSize * Add test data and sibling nodes for each node in path * Fix GetSiblingNodes * Add more test data * testing: fuzz: failing test case * Use set for keys * Add more test data * Refactor existence proofs code * Add required info for remove operation * Add children of siblings as well * Remove debug code * Add testdata that breaks current code * Fix bug * Add failing testcase * Add breaking testcase * IAVL with tracing * Fuzz tests pass * Refactor tracing code * Remove redundant code * Remove working hash in calls to AddExistenceProof * Clean up flag * Make build tree a private method * Add back whitespace in node.go * Add new ci for fuzz * Refactor more * Refactor out getKey method * Change name to reapInclusionProofs * Refactor set and remove in DST * Factor out add existence proofs from Remove DST for consistency with Set * Refactor into testContext * Clean up setInDST * Add method for get * Export methods * Add witness data to deep subtree * Verify operation in witness data * Refactor and verify operation for get and remove * Add set witness data * Add tracing to tree * add getter for witness data * Verify existence proofs in dst * Cleanup * Reset witness data on tracing enabled * Add node to keysAccessed even not in cache * Add initial root hash * Refactor GetInitialRootHash * Modify GetInitialRootHash * Add test data * Add get to dst tests: fails right now * Refactor and add tracing for root key as well * Add docs * Add more docs * Update comments * Update log message Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * allocate length Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> * fix: remove `RangeProofs` (cosmos#586) Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com> Co-authored-by: Marko <marbar3778@yahoo.com> * Update go.mod file * Add new line at end of .golangci.yml Co-authored-by: Tomasz Zdybał <tomek@zdybal.lap.pl> Co-authored-by: Matthew Sevey <mjsevey@gmail.com> Co-authored-by: cool-developer <51834436+cool-develope@users.noreply.github.com> Co-authored-by: colin axnér <25233464+colin-axner@users.noreply.github.com> Co-authored-by: Marko <marbar3778@yahoo.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposal by @jaekwon to store versions in a linked-list. The main benefit here is we don't need to store a version cache in memory, and it performs roughly the same.