Feature/Era1 import/export

[ak88]

Includes #7665 #7680 #7927

Introduces a new archive format called Era1, which is based on the Era spec made by nimbus, and this proposal from arnetheduck.

The purpose of Era1 is to provide the same functionality for EL, that Era provides for the CL. An easily verifiable and shareable format for pre-merge historic EL data, which is considered a prerequisite for eip-4444 . Era1 only concerns pre-merge data, since post-merge EL payload is contained in the beacon chain.

In the long term the idea is to distribute historic data through the portal network.

Underneath Era1 is based on e2store which is a simple type-length-value scheme.

Specification

An Era1 archive can be expressed like so

   era1 := Version | block-tuple* | other-entries* | Accumulator | BlockIndex
   block-tuple :=  CompressedHeader | CompressedBody | CompressedReceipts | TotalDifficulty
   block-index := starting-number | index | index | index ... | count

Headers, bodies and receipts are compressed using the snappy framing format.
Additionally each file contains a block index for fast lookup, and an Epoch accumulator for verification.

The epoch accumulator can be used to verify the entire archive with accumulators from a trusted source. Additionally it allows a node to download a header with a merkle-proof, that proves it belongs to a certain epoch.

The Epoch accumulator is defined in the portal network spec here.

Behaviour

• Two operation is added, import and export can be invoked via rpc or cli.

  • --Era.ImportDirectory "something" --Era.From 0 --Era.To 0 --Era.TrustedAccumulatorFile "somefile"
    • From, To and TrustedAccumulatorFile is optional. Will import everything when range is set to 0. Will trust the era directory if TrustedAccumulatorFile is not specified.
    • RPC is admin_importHistory.
  • --Era.ExportDirectory "something" --Era.From 0 --Era.To 0
    • From, To is optional. Will export everything when set to 0.
    • accumulators.txt and checksums.txt is always created.
    • RPC is admin_exportHistory.

• For simpliciy, CLI will block the application from starting until it is completed. RPC does not block but only one import/export can run at the same time.

• During import, block range before head will be inserted in parallel like old bodies, and after head will be "suggested" like forward sync. So it will process new imported block.

  • It will not complete until the blocks are processed, and will pause adding if the processing queue reaches 2000 block.
  • An exception to this is if fastsync is enabled and the head is 0, in which case, it will only insert blocks like old bodies.
  • Accumulator and content is always verified/checksumed, along with IBlockValidator.

• During export, it export the range in parallel, each era file having its own task.

Types of changes

What types of changes does your code introduce?

• New feature (a non-breaking change that adds functionality)

Testing

Requires testing

• Yes

If yes, did you write tests?

• Yes

Documentation

Requires documentation update

• Yes

Requires explanation in Release Notes

• Yes

[flcl42]

So we dispose it even if it's passed externally? And we don't need to care about double disposal?

[flcl42]

One more \n after