Upgrade in-memory and on-disk state representation with tree states #3206

michaelsproul · 2022-05-23T08:23:10Z

This is an umbrella PR for fundamental changes to Lighthouse's state representation.

The plan is to merge smaller PRs into this one, and then merge it to unstable without squashing.

TODO List

There are few things that need fixing on this branch before it's ready for primetime:

Remove recursion from get_state. The recursion is liable to overflow the stack, so we should switch to an iterative algorithm.
Fix the validator monitor
Clean up the state processing optimisations
Fuzz the state processing optimisations
Add or fix state processing optimisations for phase0 so that block replay isn't too slow
Add new metrics for jemalloc
Resolve all remaining FIXME(sproul)s in the code
Re-do the optimisation that was reverted in [Merged by Bors] - Revert "Optimise HTTP validator lookups" #3658

Issues Addressed

Closes #2806
Closes #2805
Closes #2944
Closes #2371
Closes #3208
Closes #3505

## Issue Addressed Part of #3651. ## Proposed Changes Add a flag for enabling the light client server, which should be checked before gossip/RPC traffic is processed (e.g. #3693, #3711). The flag is available at runtime from `beacon_chain.config.enable_light_client_server`. Additionally, a new method `BeaconChain::with_mutable_state_for_block` is added which I envisage being used for computing light client updates. Unfortunately its performance will be quite poor on average because it will only run quickly with access to the tree hash cache. Each slot the tree hash cache is only available for a brief window of time between the head block being processed and the state advance at 9s in the slot. When the state advance happens the cache is moved and mutated to get ready for the next slot, which makes it no longer useful for merkle proofs related to the head block. Rather than spend more time trying to optimise this I think we should continue prototyping with this code, and I'll make sure `tree-states` is ready to ship before we enable the light client server in prod (cf. #3206). ## Additional Info I also fixed a bug in the implementation of `BeaconState::compute_merkle_proof` whereby the tree hash cache was moved with `.take()` but never put back with `.restore()`.

## Issue Addressed Part of sigp#3651. ## Proposed Changes Add a flag for enabling the light client server, which should be checked before gossip/RPC traffic is processed (e.g. sigp#3693, sigp#3711). The flag is available at runtime from `beacon_chain.config.enable_light_client_server`. Additionally, a new method `BeaconChain::with_mutable_state_for_block` is added which I envisage being used for computing light client updates. Unfortunately its performance will be quite poor on average because it will only run quickly with access to the tree hash cache. Each slot the tree hash cache is only available for a brief window of time between the head block being processed and the state advance at 9s in the slot. When the state advance happens the cache is moved and mutated to get ready for the next slot, which makes it no longer useful for merkle proofs related to the head block. Rather than spend more time trying to optimise this I think we should continue prototyping with this code, and I'll make sure `tree-states` is ready to ship before we enable the light client server in prod (cf. sigp#3206). ## Additional Info I also fixed a bug in the implementation of `BeaconState::compute_merkle_proof` whereby the tree hash cache was moved with `.take()` but never put back with `.restore()`.

* Address some clippy lints * Box errors to fix error size lint * Add Default impl for Validator * Address more clippy lints * Re-implement `check_state_diff` * Fix misc test compile errors

* Use "rebasing" to minimise BeaconState mem usage * Update metastruct * Use upstream milhouse, update cargo lock * Rebase caches for extra memory savings

* Get tests passing * Get benchmarks compiling * Fix EF withdrawals test * Remove unused deps * Fix tree_hash panic in tests * Fix slasher compilation * Fix ssz_generic test * Get more tests passing * Fix EF tests for real * Fix local testnet scripts

Workaround for #4236

* Fix db-migration-period default * Fix version regex

* Relocate epoch cache to BeaconState * Optimize per block processing by pulling previous epoch & current epoch calculation up. * Revert `get_cow` change (no performance improvement) * Initialize `EpochCache` in epoch processing and load it from state when getting base rewards. * Initialize `EpochCache` at start of block processing if required. * Initialize `EpochCache` in `transition_blocks` if `exclude_cache_builds` is enabled * Fix epoch cache initialization logic * Remove FIXME comment. * Cache previous & current epochs in `consensus_context.rs`. * Move `get_base_rewards` from `ConsensusContext` to `BeaconState`. * Update Milhouse version

database_manager/src/lib.rs

dapplion · 2024-01-23T06:17:40Z

beacon_node/store/src/iter.rs

@@ -377,131 +378,3 @@ fn slot_of_prev_restore_point<E: EthSpec>(current_slot: Slot) -> Slot {
    let slots_per_historical_root = E::SlotsPerHistoricalRoot::to_u64();
    (current_slot - 1) / slots_per_historical_root * slots_per_historical_root
 }
-
-#[cfg(test)]
-mod test {


Intentional delete? BlockRootsIterator still exists

dapplion · 2024-01-23T06:19:51Z

beacon_node/store/src/leveldb_store.rs

@@ -26,6 +26,7 @@ impl<E: EthSpec> LevelDB<E> {
        let mut options = Options::new();

        options.create_if_missing = true;
+        options.write_buffer_size = Some(512 * 1024 * 1024);


Add comment justifying the increased value

consensus/state_processing/src/common/initiate_validator_exit.rs

dapplion · 2024-01-23T07:32:48Z

consensus/state_processing/src/per_block_processing/process_operations.rs

+                let epoch_participation = state.get_epoch_participation_mut(
+                    data.target.epoch,
+                    ctxt.previous_epoch,
+                    ctxt.current_epoch,


Does this optimization make a difference? Introduced in 2df714e

dapplion · 2024-01-23T07:54:57Z

consensus/types/src/beacon_state.rs

+                .map_or(false, VList::has_pending_updates)
+            || self
+                .inactivity_scores()
+                .map_or(false, VList::has_pending_updates)


Forgetting to add a field in a new fork here may cause either a panic or data corruption. This function could be generated by a macro?

dapplion · 2024-01-24T02:47:30Z

consensus/types/src/beacon_state.rs

+                    base_inner,
+                    |_, self_field, base_field| { self_field.rebase_on(base_field) }
+                );
+            }


Not using Deneb's bimap_beacon_state_deneb_tree_list_fields

* Delete `lighthouse db diff` * Fix help text

* Delete unused epoch processing code * Compare total deltas * Remove unnecessary apply_pending * cargo fmt * Remove newline

dapplion · 2024-02-08T08:46:46Z

beacon_node/store/src/hot_cold_store.rs

+                    "Updating historic state config";
+                    "previous_config" => ?disk_config.hierarchy_config,
+                    "new_config" => ?db.config.hierarchy_config,
+                );


@michaelsproul I recall you commenting changing the hierarchy_config is not possible without a re-sync. Should this comment state so?

Well if you see this comment then it has already worked, so not sure if it's helpful at this stage.

You can change it by running prune-states and then restarting with the new config. I think it probably makes sense to just have that in the docs.

* Fix beta compiler lints * Fix fork choice tests * Fix op_pool tests --------- Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com>

* Fix CLI help text * Tree states release v5.0.111-exp

Closes #5141

* Fix clippy * Fix beacon chain tests * Comment out dodgy iterator test * Fix imports for beta compiler

michaelsproul added do-not-merge work-in-progress PR is a work-in-progress backwards-incompat Backwards-incompatible API change labels May 23, 2022

michaelsproul mentioned this pull request Sep 2, 2022

Heavy HTTP requests halt the client #3536

Closed

michaelsproul mentioned this pull request Oct 4, 2022

Space-efficient archive node #3626

Merged

This was referenced Oct 19, 2022

Compress and linearise beacon blocks on disk #3583

Closed

Light client server support [tracking issue] #3651

Open

michaelsproul mentioned this pull request Oct 26, 2022

Implement per-validator reward APIs #3661

Open

michaelsproul mentioned this pull request Nov 11, 2022

[Merged by Bors] - Add --light-client-server flag and state cache utils #3714

Closed

michaelsproul mentioned this pull request Dec 2, 2022

calling /eth/v1/validator/duties/proposer/ for old epochs is very slow #3770

Closed

michaelsproul mentioned this pull request Apr 11, 2023

Optimise historic sequential state lookups #3873

Closed

michaelsproul mentioned this pull request Apr 28, 2023

[Merged by Bors] - use state cache to optimise historical state lookup #4228

Closed

Implement tree states & hierarchical state DB

23db089

michaelsproul force-pushed the tree-states branch from ba7ad3e to 23db089 Compare June 19, 2023 00:19

michaelsproul added the tree-states Upcoming state and database overhaul label Jun 19, 2023

paulhauner and others added 13 commits June 20, 2023 11:47

Address clippy lints in tree-states (#4414)

d56cec8

* Address some clippy lints * Box errors to fix error size lint * Add Default impl for Validator * Address more clippy lints * Re-implement `check_state_diff` * Fix misc test compile errors

Configurable diff buffer cache size (#4420)

6eb1513

Use rebasing to minimise BeaconState mem usage (#4416)

ca412ab

* Use "rebasing" to minimise BeaconState mem usage * Update metastruct * Use upstream milhouse, update cargo lock * Rebase caches for extra memory savings

Fix failing tests (#4423)

88e30b6

* Get tests passing * Get benchmarks compiling * Fix EF withdrawals test * Remove unused deps * Fix tree_hash panic in tests * Fix slasher compilation * Fix ssz_generic test * Get more tests passing * Fix EF tests for real * Fix local testnet scripts

Set epochs per migration to 1

7c2eb96

Workaround for #4236

Install Clang 5 in Cross builder image

56c7a52

Tree states alpha release v4.2.99-exp

af5fb20

Temporarily disable ARM builds

8dc374e

Tree states alpha release v4.2.990-exp

6954de6

Fix db-migration-period default (#4441)

160bbde

* Fix db-migration-period default * Fix version regex

Fix EpochCache handling in ef-tests

f631b51

Fix EpochCache handling in ef-tests (#4454)

835fa70

dapplion reviewed Jan 25, 2024

View reviewed changes

michaelsproul added 6 commits January 30, 2024 09:59

Fix new CLI tests for tree-states (#5132)

11461d8

Fix tree-states sub-epoch diffs (#5097)

7862c71

fsync during backfill to prevent DB corruption (#5144)

8e68926

Merge remote-tracking branch 'origin/unstable' into tree-states

68a9a2e

Tree states v4.6.222-exp (#5147)

25bcd2a

Delete lighthouse db diff (#5171)

ae6620e

* Delete `lighthouse db diff` * Fix help text

dapplion mentioned this pull request Feb 7, 2024

Epoch single pass dapplion/lighthouse#13

Closed

4 tasks

Delete unused epoch processing code (#5170)

8e2f7c5

* Delete unused epoch processing code * Compare total deltas * Remove unnecessary apply_pending * cargo fmt * Remove newline

dapplion reviewed Feb 8, 2024

View reviewed changes

michaelsproul and others added 5 commits February 14, 2024 11:55

Use epoch cache in block packing (#5223)

9eb1685

Remove progressive balances mode (#5224)

b1f7f0a

Merge remote-tracking branch 'origin/unstable' into tree-states

26117a5

Fix tree-states tests (#5277)

f9c9c40

* Fix beta compiler lints * Fix fork choice tests * Fix op_pool tests --------- Co-authored-by: dapplion <35266934+dapplion@users.noreply.github.com>

Tree states release v5.0.111-exp (#5276)

20f53e7

* Fix CLI help text * Tree states release v5.0.111-exp

michaelsproul mentioned this pull request Feb 23, 2024

Single-pass epoch processing and optimised block processing #5279

Merged

dapplion added 3 commits February 23, 2024 11:49

Merge remote-tracking branch 'sigp/epoch-single-pass' into tree-states

a5d3408

Remove milhouse export aliases (#5286)

5dfc5c1

Closes #5141

as_store_bytes not fallible (#5285)

928915c

michaelsproul mentioned this pull request Mar 7, 2024

SSV use lighthouse v5.0.0， proposal block miss. #5365

Closed

Merge remote-tracking branch 'origin/unstable' into tree-states-update

a249dbf

mergify bot mentioned this pull request Mar 11, 2024

Update tree-states branch with changes from v5.1.0 #5388

Merged

michaelsproul added 3 commits March 12, 2024 10:36

Tree states release v5.1.111-exp

03e2a39

Fix macOS build by bumping cc (#5392)

a9f4457

Tree states release v5.1.222-exp

2cd0e60

michaelsproul mentioned this pull request Mar 27, 2024

Fork choice should run after RPC blob processing #5474

Closed

michaelsproul added 3 commits April 5, 2024 15:14

Merge remote-tracking branch 'origin/unstable' into tree-states

0d9fb5a

Fix tree-states tests after latest update (#5528)

8705ac5

* Fix clippy * Fix beacon chain tests * Comment out dodgy iterator test * Fix imports for beta compiler

Resolve rebase_on issues

b5f2761

michaelsproul mentioned this pull request Apr 8, 2024

In-memory tree states #5533

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade in-memory and on-disk state representation with tree states #3206

Upgrade in-memory and on-disk state representation with tree states #3206

michaelsproul commented May 23, 2022 •

edited

Loading

dapplion Jan 23, 2024

dapplion Jan 23, 2024

dapplion Jan 23, 2024

dapplion Jan 23, 2024

dapplion Jan 24, 2024

dapplion Feb 8, 2024

michaelsproul Feb 8, 2024

Upgrade in-memory and on-disk state representation with tree states #3206

Are you sure you want to change the base?

Upgrade in-memory and on-disk state representation with tree states #3206

Conversation

michaelsproul commented May 23, 2022 • edited Loading

TODO List

Issues Addressed

dapplion Jan 23, 2024

Choose a reason for hiding this comment

dapplion Jan 23, 2024

Choose a reason for hiding this comment

dapplion Jan 23, 2024

Choose a reason for hiding this comment

dapplion Jan 23, 2024

Choose a reason for hiding this comment

dapplion Jan 24, 2024

Choose a reason for hiding this comment

dapplion Feb 8, 2024

Choose a reason for hiding this comment

michaelsproul Feb 8, 2024

Choose a reason for hiding this comment

michaelsproul commented May 23, 2022 •

edited

Loading