New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP]: improve snapshot generation #22465

Closed

rjl493456442 wants to merge 28 commits into ethereum:master from rjl493456442:fill-snap

Member

rjl493456442 commented Mar 9, 2021 •

edited

Loading

Implements https://gist.github.com/rjl493456442/85832a0c760f2bafe2a69e33efe68c60, a better snapshot generation mechanism.

The range-proving based snapshot generation can be used in lots of different purposes.

It can speed up the initial snapshot generation after the snap sync, by using the existent flat state received during the sync
It can do a quick snapshot rebuild(after the corruption for example) by using the existent flat state persisted previously

So that the snapshot UX can be improved a bit.

rjl493456442 added 4 commits

March 9, 2021 13:11


          eth/protocols: persist received state segments

4a7becb


          core: initial implementation

60f57c9


          core/state/snapshot: add tests

bf6c7da


          core, eth: updates

fd7cf4b

rjl493456442 requested review from holiman and karalabe as code owners

March 9, 2021 07:23

rjl493456442 added 4 commits

March 9, 2021 15:34


          eth/protocols/snapshot: count flat state size

8bf2032


          core/state: add metrics

353804d


          core/state/snapshot: skip unnecessary deletion

3a3d341


          core/state/snapshot: rename

f0e822e

Contributor

holiman commented Mar 9, 2021

This is a neat idea, and I'm sure we'll iterate on it a bit.
One thing that we might want to add later (but maybe first get the basic version working), is to also update the snapshot layer when we do trie healing, for any accounts that are updated. Theoretically, we will then get a pristine snapshot layer, where the only errors are due to

accounts that were deleted during the sync
storage slots that were cleared during the sync (I think)

I'd expect this to decrease the error-margin of the snap layer probably by an order of magnitude or more.

holiman reviewed

View reviewed changes

core/state/snapshot/wipe.go Show resolved Hide resolved

rjl493456442 added 8 commits

March 9, 2021 20:36


          core/state/snapshot: use the global batch

0b8d5a6


          core/state/snapshot: add logs and fix wiping

a9ea5be


          core/state/snapshot: fix

0a2a1a8


          core/state/snapshot: save generation progress even if the batch is empty

b6b2aaf


          core/state/snapshot: fixes

f64ede8


          core/state/snapshot: fix initial account range length

e848ac9


          core/state/snapshot: fix initial account range

f5b9a8f


          eth/protocols/snap: store flat states during the healing

cc8bffc

holiman reviewed

View reviewed changes

core/state/snapshot/generate.go Outdated Show resolved Hide resolved

core/state/snapshot/generate.go Outdated Show resolved Hide resolved

core/state/snapshot/generate.go

-              	accTrie, err := trie.NewSecure(dl.root, dl.triedb)
+              	// Verify the state segment with range prover, ensure that all flat states
+              	// in this range correspond to merkle trie.
+              	_, _, _, cont, err := trie.VerifyRangeProof(root, origin, last, keys, vals, proof)

Contributor

holiman Mar 11, 2021

Since you ignore the db here, it would be neat to use a version of the range prover which uses the stacktrie instead of the full trie.
OTOH it doesn't matter a whole lot, since you're only using a few hundred key/values per iteration

Member Author

rjl493456442 Mar 12, 2021

Actually the stacktrie is not suitable for range prove. Stacktrie expects all the inputs from the most left side and to the most right side. It won't accept the segment in the middle, even the segment is continuous.

core/state/snapshot/generate.go Outdated Show resolved Hide resolved

holiman reviewed

View reviewed changes

eth/protocols/snap/sync.go Outdated Show resolved Hide resolved

rjl493456442 and others added 4 commits

March 12, 2021 11:05


          eth/protocols/snap: print logs

475ad07


          core/state/snapshot: refactor (#4)

955e99d

* core/state/snapshot: refactor

* core/state/snapshot: tiny fix and polish

Co-authored-by: rjl493456442 <garyrong0905@gmail.com>


          core, eth: fixes

27c8239


          core, eth: fix healing writer

a8c419a

rjl493456442 mentioned this pull request

Snapshot relevant tools #22264

Closed

rjl493456442 added 2 commits

March 12, 2021 15:33


          core, trie, eth: fix paths

523e3e4


          eth/protocols/snap: fix encoding

3f992cf

holiman reviewed

View reviewed changes

core/state/snapshot/generate.go Outdated Show resolved Hide resolved

holiman mentioned this pull request

core/state/generate: release iterator asap rjl493456442/go-ethereum#5

Merged

rjl493456442 and others added 6 commits

March 15, 2021 10:29


          eth, core: add debug log

9f27aae


          core/state/generate: release iterator asap (#5)

8344c7c

core/state/snapshot: less copy

core/state/snapshot: revert split loop

core/state/snapshot: handle storage becoming empty, improve test robustness

core/state: test modified codehash

core/state/snapshot: polish


          core/state/snapshot: optimize stats counter

e69cde1


          core, eth: add metric

f52d630


          core/state/snapshot: update comments

2dd3d99


          core/state/snapshot: improve tests

b77a7ff

holiman reviewed

View reviewed changes

core/state/snapshot/generate.go Show resolved Hide resolved

holiman mentioned this pull request

WIP 2 Improve snapshot generation #22500

Closed

Contributor

holiman commented Mar 18, 2021

Can we close this in favour of #22504 ?

holiman closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet