Not able to download finalized state

[twoeths]

Describe the bug

Downloading a finalized slot on mainnet: curl -H "Accept: application/octet-stream" http://127.0.0.1:9596/eth/v2/debug/beacon/states/7024415 -o mainnet_state_7024415.ssz

and it returned

{"statusCode":404,"error":"Not Found","message":"No state found for id '7024415'"}

if I download by state root: curl -H "Accept: application/octet-stream" http://127.0.0.1:9596/eth/v2/debug/beacon/states/0xd27419094c269f3f1069b0dbe754eccb9e073ab6023cf8cf910eb07e29866723 -o mainnet_state_7024415.ssz

I got

{"statusCode":500,"error":"Internal Server Error","message":"REGEN_ERROR_NO_SEED_STATE"}

Expected behavior

should be able to download state

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.9.2

[twoeths]

a regular lodestar node just store some finalized epochs every 32 or 1024 epochs, also regen does not support for finalized slot

[yrong]

Is there any way to increase the cached size for beacon states?

[nflaig]

Is there any way to increase the cached size for beacon states?

What exactly do you mean by cached size?

The max number of states allowed in the cache is 96 right now, this is not configurable but this value should also not be relevant for end users.

lodestar/packages/beacon-node/src/chain/stateCache/stateContextCache.ts

Line 8 in d988974

const MAX_STATES = 3 * 32;

Finalized states are pruned and can no longer be retrieved from the cache, irrespective of the max number of allowed states.

lodestar/packages/beacon-node/src/chain/archiver/index.ts

Line 106 in d988974

this.chain.regen.pruneOnFinalized(finalizedEpoch);

If you want to be able to query finalized states from Lodestar you have to set --chain.archiveStateEpochFrequency 1. This would allow to query the state of the start slot in the epoch (epoch * 32) but your disk requirements will increase significantly.

While looking at this I also noticed this only persists the state every second epoch, will be fixed by #5979.

[twoeths]

according to this, all other clients support regenerating a finalized state while lodestar is not https://discord.com/channels/593655374469660673/1062235362509996112/1161371178728968244

at least I see lighthouse does this https://github.com/sigp/lighthouse/blob/v4.5.0/beacon_node/http_api/src/state_id.rs#L194

one important thing to note is other clients may handle http requests in a separate thread while lodestar is not and this request could blow up the node, we may need to consider offloading this work to a worker thread in order not to affect the node

[wemeetagain]

resolved with #6033