Snapshot relevant tools

[rjl493456442]

Snapshot is an important component used in Geth, which is basically the flatten format state for fast access and iteration.
However, the snapshot generation is quite slow(generated by the trie iteration) and it's fragile. It's really annoying to re-generate the snapshot after some exceptions.

This issue proposes a tool to recover the corrupted snapshot, so that we can avoid the regeneration in most cases.

The tool can be used for verifying the correctness of the snapshot, especially the disk layer. Local merkle trie and range prover can be used to detect the corrupted range in the disk layer(if the merkle trie is not corrupted) and fix it.

The tool can be used to run the generation or wipe the snapshot offline(probably it's faster than online version)

And also the tool can be used to apply the state transitions based on a complete snapshot. It's very common that the user enables the snapshot previously and later forget to enable it again. So the gapped snapshot is entirely useless. The tool can fix the gap by executing the relevant blocks.

[rjl493456442]

A few more ideas:

The tool can be used for verifying the correctness of the snapshot, especially the disk layer. Local merkle trie and range prover can be used to detect the corrupted range in the disk layer(if the merkle trie is not corrupted) and fix it.

This can be integrated into the current generation/rebuilding procedure seamlessly. In the following scenarios:

• The disk layer is just snap-synced and need the correction
• The disk layer is broken with a part of the segment corrupted
• The snapshot is building from the scratch(migrate from the old Geth version)
  The assumption can be held that: the flat data exists in the disk and maybe can be reused. An additional operation is added during the generation:
• Iterate the flat data from the database in batch(for example, iterate out 100 items)
• Construct the range proof for these items with proper edge paths
• Verify the correctness of this particular range
• Wipe the range if it's failed to range proof and regenerate from the trie

The overhead is the sequential iteration for flat data and runs the range proof(it's kind of cheap compared with trie iteration). The benefit is we can skip most of the regenration(trie iteration is expensive) and reuse most of the existent data.

For the scenario, we are building the snapshot from the scratch, we can add some tricks by skipping the prechecks.

[rjl493456442]

The entire idea is implemented in #22465 :)

[holiman]

I think we have this now (at least implicitly, if not as explicit command line commands) -- can we close it?

[rjl493456442]

YES!