Compress and linearise beacon blocks on disk #3583
Conversation
Squashed commit of the following: commit a4960eb Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Sep 12 12:10:23 2022 +1000 Clippy commit b28e8d0 Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Sep 12 11:41:45 2022 +1000 Add flag to disable prune on startup commit de775d6 Author: Michael Sproul <michael@sigmaprime.io> Date: Mon Sep 12 11:19:21 2022 +1000 Fix and update beacon chain tests commit 2289b20 Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 9 17:40:21 2022 +1000 Implement DB manager command commit d5adc2e Author: Michael Sproul <michael@sigmaprime.io> Date: Fri Sep 9 12:56:27 2022 +1000 Implement on-demand pruning operation commit 69d5474 Author: Michael Sproul <michael@sigmaprime.io> Date: Thu Sep 8 16:25:04 2022 +1000 Delete finalized exec payloads while running
|
Syncing a new node with this code to see a) how big the database ends up and b) what the I/O pattern is like once backfill completes. |
|
This is looking good from a disk usage point of view, using only 48.1 GB after a few days (less than half what we currently use). However the impact on I/O write amplification is not as good as I'd hoped, with only a ~10% reduction in gigabytes written per day. Data here: https://docs.google.com/spreadsheets/d/1-5ewstEFOQzHpoMLmlpWBBHBtJ7klSrjaVSmqwV7t0o/edit?usp=sharing. I think the remaining writes are probably coming from hot states, which we write every epoch boundary. Using |
|
Closing as this is now merged into #3206. |
Issue Addressed
Closes #3208
Proposed Changes
Very much WIP but this seems to be not too bad. Main TODOs are testing performance and providing backwards compatibility