[Feature]: use plain files to store snapshots

[yihuang]

Summary

use plain files to store snapshots for easier out-of-band snapshot syncing

Problem Definition

Right now we store the chunks in files, but snapshot metadata is stored in a kv db, change to plain files will make it easier for external tools to manipulate the snapshots, which will be useful when we use local snapshot restorations and sync the snapshots out-of-band.

Proposal

Just use plain files and directories to store snapshots, without kv db.

- snapshots
  - 00000000000008190000-002 # <height>-<format> padding with enough zeroes to keep the ordering
                           # this directory will be very easy to move around.
    - snapshot             # protobuf serialized `abci.RequestOfferSnapshot`
    - chunks
      - 1
      - 2

• Use os.ReadDir to iterate the snapshots
• Read the snapshot file to get snapshot metadata
• Chunk files are the same as before
• Less performant than kv db, but should be alright since the number of snapshots won't be too big.

[yihuang]

A simpler alternative is just a command that import/export the snapshot in a portable format, basically just bundle the Snapshot message together with the chunk files.

[alexanderbez]

@tac0turtle didn't you work on a tool to do this? In any case, I think the proposal makes sense.

[yihuang]

there are two caveats though:

• change existing snapshots directory format, so need to handle the migration.
• need to be careful of atomicity of the operations, otherwise ListSnapshot could discover an incomplete snapshot.

[yihuang]

I'll close this with #16067, it should be good enough for now with these management commands.