Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

core/state/snapshot: faster snapshot generation #22504

Merged
merged 75 commits into from
Apr 14, 2021
Merged

core/state/snapshot: faster snapshot generation #22504

Show file tree
Hide file tree
Changes from 72 commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
4c2c35f
eth/protocols: persist received state segments
rjl493456442 Mar 5, 2021
4a7650f
core: initial implementation
rjl493456442 Mar 8, 2021
2b65233
core/state/snapshot: add tests
rjl493456442 Mar 9, 2021
d74a7f1
core, eth: updates
rjl493456442 Mar 9, 2021
5a26f7f
eth/protocols/snapshot: count flat state size
rjl493456442 Mar 9, 2021
d21e5a2
core/state: add metrics
rjl493456442 Mar 9, 2021
65cee53
core/state/snapshot: skip unnecessary deletion
rjl493456442 Mar 9, 2021
557b81c
core/state/snapshot: rename
rjl493456442 Mar 9, 2021
45fead8
core/state/snapshot: use the global batch
rjl493456442 Mar 9, 2021
a9bea12
core/state/snapshot: add logs and fix wiping
rjl493456442 Mar 10, 2021
310db7c
core/state/snapshot: fix
rjl493456442 Mar 10, 2021
6102044
core/state/snapshot: save generation progress even if the batch is empty
rjl493456442 Mar 10, 2021
dd4ced3
core/state/snapshot: fixes
rjl493456442 Mar 10, 2021
5eb07d4
core/state/snapshot: fix initial account range length
rjl493456442 Mar 10, 2021
62eb856
core/state/snapshot: fix initial account range
rjl493456442 Mar 11, 2021
b2dbe3b
eth/protocols/snap: store flat states during the healing
rjl493456442 Mar 11, 2021
529147f
eth/protocols/snap: print logs
rjl493456442 Mar 12, 2021
da8f26b
core/state/snapshot: refactor (#4)
holiman Mar 12, 2021
7433151
core, eth: fixes
rjl493456442 Mar 12, 2021
7073457
core, eth: fix healing writer
rjl493456442 Mar 12, 2021
70c9b32
core, trie, eth: fix paths
rjl493456442 Mar 12, 2021
d865c47
eth/protocols/snap: fix encoding
rjl493456442 Mar 12, 2021
6e94768
eth, core: add debug log
rjl493456442 Mar 15, 2021
afecd67
core/state/generate: release iterator asap (#5)
holiman Mar 15, 2021
9cf7677
core/state/snapshot: optimize stats counter
rjl493456442 Mar 15, 2021
13d4b9b
core, eth: add metric
rjl493456442 Mar 15, 2021
0cceb3c
core/state/snapshot: update comments
rjl493456442 Mar 15, 2021
1f05908
core/state/snapshot: improve tests
rjl493456442 Mar 15, 2021
07e3575
core/state/snapshot: replace secure trie with standard trie
rjl493456442 Mar 16, 2021
1d3517e
core/state/snapshot: wrap return as the struct
rjl493456442 Mar 16, 2021
7c8e43c
core/state/snapshot: skip wiping correct states
rjl493456442 Mar 16, 2021
4539a8f
core/state/snapshot: updates
rjl493456442 Mar 16, 2021
240e9d6
core/state/snapshot: fixes
rjl493456442 Mar 16, 2021
2fd4ce2
core/state/snapshot: fix panic due to reference flaw in closure
holiman Mar 16, 2021
b221f1a
core/state/snapshot: fix errors in state generation logic + fix log o…
holiman Mar 16, 2021
6c93d5a
core/state/snapshot: remove an error case
holiman Mar 16, 2021
2cf335c
core/state/snapshot: fix condition-check for exhausted snap state
holiman Mar 16, 2021
f25852e
core/state/snapshot: use stackTrie for small tries
holiman Mar 16, 2021
dab43b3
core/state/snapshot: don't resolve small storage tries in vain
holiman Mar 16, 2021
2fce60d
core/state/snapshot: properly clean up storage of deleted accounts
holiman Mar 16, 2021
148bde6
core/state/snapshot: avoid RLP-encoding in some cases + minor nitpicks
holiman Mar 16, 2021
c24ef63
core/state/snapshot: fix error (+testcase)
holiman Mar 16, 2021
2625274
core/state/snapshot: clean up tests a bit
holiman Mar 16, 2021
ac645a1
core/state/snapshot: work in progress on better tests
holiman Mar 16, 2021
823d0c5
core/state/snapshot: polish code
rjl493456442 Mar 17, 2021
5a36506
core/state/snapshot: fix trie iteration abortion trigger
rjl493456442 Mar 17, 2021
664615f
core/state/snapshot: fixes flaws
rjl493456442 Mar 17, 2021
95118e7
core/state/snapshot: remove panic
rjl493456442 Mar 17, 2021
bba0f1c
core/state/snapshot: fix abort
rjl493456442 Mar 17, 2021
3c64389
core/state/snapshot: more tests (plus failing testcase)
holiman Mar 17, 2021
47bc0ef
core/state/snapshot: more testcases + fix for failing test
holiman Mar 17, 2021
71ea5ad
core/state/snapshot: testcase for malformed data
holiman Mar 17, 2021
63f4998
core/state/snapshot: some test nitpicks
holiman Mar 17, 2021
ac8ebf3
core/state/snapshot: improvements to logging
holiman Mar 17, 2021
fda1c2a
core/state/snapshot: testcase to demo error in abortion
holiman Mar 18, 2021
f58f3c7
core/state/snapshot: fix abortion
rjl493456442 Mar 18, 2021
ea540aa
cmd/geth: make verify-state report the root
holiman Mar 18, 2021
37b3426
trie: fix failing test
holiman Mar 18, 2021
d172f80
core/state/snapshot: add timer metrics
rjl493456442 Mar 18, 2021
3c9073c
core/state/snapshot: fix metrics
rjl493456442 Mar 19, 2021
1e42318
core/state/snapshot: udpate tests
rjl493456442 Mar 19, 2021
4563da2
eth/protocols/snap: write snapshot account even if code or state is n…
holiman Mar 26, 2021
c78f9f4
core/state/snapshot: fix diskmore check
holiman Mar 26, 2021
c27ea4f
core/state/snapshot: review fixes
holiman Mar 26, 2021
46fc218
core/state/snapshot: improve error message
rjl493456442 Mar 31, 2021
bb149d9
cmd/geth: rename 'error' to 'err' in logs
holiman Apr 14, 2021
78cdadd
core/state/snapshot: fix some review concerns
holiman Apr 14, 2021
1693ca3
core/state/snapshot, eth/protocols/snap: clear snapshot marker when s…
holiman Apr 14, 2021
c2e7952
core: add error log
rjl493456442 Apr 14, 2021
1cc3ca3
core/state/snapshot: use proper timers for metrics collection
holiman Apr 14, 2021
1a6143f
core/state/snapshot: address some review concerns
holiman Apr 14, 2021
920e2eb
eth/protocols/snap: improved log message
holiman Apr 14, 2021
fa8b467
eth/protocols/snap: fix heal logs to condense infos
karalabe Apr 14, 2021
3a54830
core/state/snapshot: wait for generator termination before restarting
karalabe Apr 14, 2021
12a1a85
core/state/snapshot: revert timers to counters to track total time
karalabe Apr 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 19 additions & 19 deletions cmd/geth/snapshot.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ func pruneState(ctx *cli.Context) error {
chaindb := utils.MakeChainDatabase(ctx, stack, false)
pruner, err := pruner.NewPruner(chaindb, stack.ResolvePath(""), stack.ResolvePath(config.Eth.TrieCleanCacheJournal), ctx.GlobalUint64(utils.BloomFilterSizeFlag.Name))
if err != nil {
log.Error("Failed to open snapshot tree", "error", err)
log.Error("Failed to open snapshot tree", "err", err)
return err
}
if ctx.NArg() > 1 {
Expand All @@ -166,12 +166,12 @@ func pruneState(ctx *cli.Context) error {
if ctx.NArg() == 1 {
targetRoot, err = parseRoot(ctx.Args()[0])
if err != nil {
log.Error("Failed to resolve state root", "error", err)
log.Error("Failed to resolve state root", "err", err)
return err
}
}
if err = pruner.Prune(targetRoot); err != nil {
log.Error("Failed to prune state", "error", err)
log.Error("Failed to prune state", "err", err)
return err
}
return nil
Expand All @@ -189,7 +189,7 @@ func verifyState(ctx *cli.Context) error {
}
snaptree, err := snapshot.New(chaindb, trie.NewDatabase(chaindb), 256, headBlock.Root(), false, false, false)
if err != nil {
log.Error("Failed to open snapshot tree", "error", err)
log.Error("Failed to open snapshot tree", "err", err)
return err
}
if ctx.NArg() > 1 {
Expand All @@ -200,15 +200,15 @@ func verifyState(ctx *cli.Context) error {
if ctx.NArg() == 1 {
root, err = parseRoot(ctx.Args()[0])
if err != nil {
log.Error("Failed to resolve state root", "error", err)
log.Error("Failed to resolve state root", "err", err)
return err
}
}
if err := snaptree.Verify(root); err != nil {
log.Error("Failed to verfiy state", "error", err)
log.Error("Failed to verfiy state", "root", root, "err", err)
return err
}
log.Info("Verified the state")
log.Info("Verified the state", "root", root)
return nil
}

Expand Down Expand Up @@ -236,7 +236,7 @@ func traverseState(ctx *cli.Context) error {
if ctx.NArg() == 1 {
root, err = parseRoot(ctx.Args()[0])
if err != nil {
log.Error("Failed to resolve state root", "error", err)
log.Error("Failed to resolve state root", "err", err)
return err
}
log.Info("Start traversing the state", "root", root)
Expand All @@ -247,7 +247,7 @@ func traverseState(ctx *cli.Context) error {
triedb := trie.NewDatabase(chaindb)
t, err := trie.NewSecure(root, triedb)
if err != nil {
log.Error("Failed to open trie", "root", root, "error", err)
log.Error("Failed to open trie", "root", root, "err", err)
return err
}
var (
Expand All @@ -262,21 +262,21 @@ func traverseState(ctx *cli.Context) error {
accounts += 1
var acc state.Account
if err := rlp.DecodeBytes(accIter.Value, &acc); err != nil {
log.Error("Invalid account encountered during traversal", "error", err)
log.Error("Invalid account encountered during traversal", "err", err)
return err
}
if acc.Root != emptyRoot {
storageTrie, err := trie.NewSecure(acc.Root, triedb)
if err != nil {
log.Error("Failed to open storage trie", "root", acc.Root, "error", err)
log.Error("Failed to open storage trie", "root", acc.Root, "err", err)
return err
}
storageIter := trie.NewIterator(storageTrie.NodeIterator(nil))
for storageIter.Next() {
slots += 1
}
if storageIter.Err != nil {
log.Error("Failed to traverse storage trie", "root", acc.Root, "error", storageIter.Err)
log.Error("Failed to traverse storage trie", "root", acc.Root, "err", storageIter.Err)
return storageIter.Err
}
}
Expand All @@ -294,7 +294,7 @@ func traverseState(ctx *cli.Context) error {
}
}
if accIter.Err != nil {
log.Error("Failed to traverse state trie", "root", root, "error", accIter.Err)
log.Error("Failed to traverse state trie", "root", root, "err", accIter.Err)
return accIter.Err
}
log.Info("State is complete", "accounts", accounts, "slots", slots, "codes", codes, "elapsed", common.PrettyDuration(time.Since(start)))
Expand Down Expand Up @@ -326,7 +326,7 @@ func traverseRawState(ctx *cli.Context) error {
if ctx.NArg() == 1 {
root, err = parseRoot(ctx.Args()[0])
if err != nil {
log.Error("Failed to resolve state root", "error", err)
log.Error("Failed to resolve state root", "err", err)
return err
}
log.Info("Start traversing the state", "root", root)
Expand All @@ -337,7 +337,7 @@ func traverseRawState(ctx *cli.Context) error {
triedb := trie.NewDatabase(chaindb)
t, err := trie.NewSecure(root, triedb)
if err != nil {
log.Error("Failed to open trie", "root", root, "error", err)
log.Error("Failed to open trie", "root", root, "err", err)
return err
}
var (
Expand Down Expand Up @@ -368,13 +368,13 @@ func traverseRawState(ctx *cli.Context) error {
accounts += 1
var acc state.Account
if err := rlp.DecodeBytes(accIter.LeafBlob(), &acc); err != nil {
log.Error("Invalid account encountered during traversal", "error", err)
log.Error("Invalid account encountered during traversal", "err", err)
return errors.New("invalid account")
}
if acc.Root != emptyRoot {
storageTrie, err := trie.NewSecure(acc.Root, triedb)
if err != nil {
log.Error("Failed to open storage trie", "root", acc.Root, "error", err)
log.Error("Failed to open storage trie", "root", acc.Root, "err", err)
return errors.New("missing storage trie")
}
storageIter := storageTrie.NodeIterator(nil)
Expand All @@ -397,7 +397,7 @@ func traverseRawState(ctx *cli.Context) error {
}
}
if storageIter.Error() != nil {
log.Error("Failed to traverse storage trie", "root", acc.Root, "error", storageIter.Error())
log.Error("Failed to traverse storage trie", "root", acc.Root, "err", storageIter.Error())
return storageIter.Error()
}
}
Expand All @@ -416,7 +416,7 @@ func traverseRawState(ctx *cli.Context) error {
}
}
if accIter.Error() != nil {
log.Error("Failed to traverse state trie", "root", root, "error", accIter.Error())
log.Error("Failed to traverse state trie", "root", root, "err", accIter.Error())
return accIter.Error()
}
log.Info("State is complete", "nodes", nodes, "accounts", accounts, "slots", slots, "codes", codes, "elapsed", common.PrettyDuration(time.Since(start)))
Expand Down
2 changes: 1 addition & 1 deletion core/state/snapshot/conversion.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -322,7 +322,7 @@ func generateTrieRoot(db ethdb.KeyValueWriter, it Iterator, account common.Hash,
return
}
if !bytes.Equal(account.Root, subroot.Bytes()) {
results <- fmt.Errorf("invalid subroot(%x), want %x, got %x", it.Hash(), account.Root, subroot)
results <- fmt.Errorf("invalid subroot(path %x), want %x, have %x", hash, account.Root, subroot)
return
}
results <- nil
Expand Down
Loading