Missing chunk in forwards iterator

[xrchz]

Description

I am occasionally seeing logs like this:

Aug 08 01:55:38.172 ERRO Missing chunk in forwards iterator, chunk index: 21318, service: freezer_db, module: store::chunked_iter:112
...
Aug 08 01:56:03.256 ERRO Missing chunk in forwards iterator, chunk index: 21318, service: freezer_db, module: store::chunked_iter:112
...
Aug 08 01:56:04.874 ERRO Missing chunk in forwards iterator, chunk index: 21314, service: freezer_db, module: store::chunked_iter:112
...
Aug 11 04:15:03.247 ERRO Missing chunk in forwards iterator, chunk index: 21304, service: freezer_db, module: store::chunked_iter:112

This started happening after I turned on the --builder option and restarted the beacon node (on Görli) -- the option may be a coincidence.

Version

Lighthouse v2.5.1-df51a73

Present Behaviour

Errors are being logged

Expected Behaviour

These errors shouldn't happen

Steps to resolve

Unknown

[xrchz]

On Discord @michaelsproul suggested this might be related to #3433

[michaelsproul]

@xrchz Can you please post the output of curl "http://localhost:5052/lighthouse/database/info?

It might be that because of checkpoint sync your node lacks those chunks, and we're just failing to check that before trying to load them

[xrchz]

{"schema_version":11,"config":{"slots_per_restore_point":8192,"slots_per_restore_point_set_explicitly":false,"block_cache_size":5,"compact_on_init":false,"compact_on_prune":true},"split":{"slot":"3640768","state_root":"0x10917822f8e0a7622f4d707626c3dfefc2b9a2afbbeb6b045eec67efaf6016f0"},"anchor":{"anchor_slot":"3284480","oldest_block_slot":"0","oldest_block_parent":"0x0000000000000000000000000000000000000000000000000000000000000000","state_upper_limit":"3284992","state_lower_limit":"0"}}

[michaelsproul]

Based on that it seems like your node should have all blocks and therefore all block roots. Can you please DM me your full debug logs (from $datadir/beacon/logs) on Discord?

[michaelsproul]

Added some debugging tools in this PR which might help us get an idea of what happened: #3511