core/state/snapshot: less disk reads #6
Conversation
* core/state/snapshot: refactor * core/state/snapshot: tiny fix and polish Co-authored-by: rjl493456442 <garyrong0905@gmail.com>
core/state/snapshot: less copy core/state/snapshot: revert split loop core/state/snapshot: handle storage becoming empty, improve test robustness core/state: test modified codehash core/state/snapshot: polish
|
Generating without this PR: ^ Note that the two runs are on the same machine at different times, and the latter image shows a much shorter time segment |
core/state/snapshot/generate.go
Outdated
| var ( | ||
| trieMore bool | ||
| iter = trie.NewIterator(tr.NodeIterator(origin)) | ||
| nodeIt = tr.NodeIterator(origin) |
There was a problem hiding this comment.
I think we can pass the auxiliary database here. iter = trie.NewIterator(tr.NodeIteratorWithDatabase(origin))
core/state/snapshot/generate.go
Outdated
| @@ -428,6 +445,7 @@ func (dl *diskLayer) generateRange(root common.Hash, prefix []byte, kind string, | |||
| istart time.Time | |||
| internal time.Duration | |||
| ) | |||
| nodeIt.AddResolver(snapTrieDb) | |||
There was a problem hiding this comment.
We can pass the db directly when we construct the iterator
Yeah, well, one thing that would be kind of neat, or at least interesting to check, is to
|
Yes sure, I can do it. We need to modify the range prover a bit, let me see how to fix it. |
rjl493456442
force-pushed
the
fill-snap-exp
branch
from
9c7813d
to
46fc218
Compare
|
Closing this in favour of ethereum#22667 |
[R4R] modify params for Parlia consensus with 21 validators
This PR attempts to minimize the number of disk reads. When we have a slice of snapshot values, which do not match the trie data, we currently iterates the disk to retrieve the canonical data.
What this PR does, is commit the (incorrect) snapshot data to a trie, which will be 99% correct. When iterating the trie, we then use the snapshot-trie-database for resolving hashes.
When doing so, we can read 99% of the leaves the from the memory db instead of resolving from disk.
This is still work in progress, needs to be tidied up. It can also be implemented differently. The upside of this particular way to implement it is that most of the modifications are performed in the node iterator, and does not touch the trie database.