[Execution Node] EN startup time can be further reduced by reducing number of WAL segments

[fxamacker]

Problem

EN startup time can be improved.

Currently, checkpoint creation is triggered when more than 40 WAL files accumulate. Replaying each WAL file can take very roughly about 10-20 seconds each (depending on system specs, other activity, etc.)

Prior to PR #1944, it wasn't feasible to create checkpoints more frequently because checkpoint creation took 12-15+ hours on mainnet. After PR #1944, checkpoint creation should take very roughly around 15 minutes while consuming much less RAM.

EN startup time can be faster by having fewer WALs to replay during EN startup.

The Proposed Solution

• Compare the benchmarks (checkpoint file loading + replaying WALS) using 40 WALs vs 20 WALs.
• If needed, configure EN to use a different number of WALs to trigger checkpoint creation.

Preliminary Benchmarks

On benchnet, using loading checkpoint and replaying WALs takes about:

• 10 mins 10 secs (40 WALs, current configuration)
• 6 mins 40 secs (20 WALs, possible configuration)

Updates https://github.com/dapperlabs/flow-go/issues/6114