failed to get logs for block #7950923 (0xff68a7..396b6d) on long running node

[shammahalchemy]

System information

Geth version: ethereum/client-go:v1.14.5
CL client & version: prysmaticlabs/prysm-beacon-chain:v5.0.4
OS & Version: Linux

Expected behaviour

Output block logs.

Actual behaviour

{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"failed to get logs for block #7950923 (0xff68a7..396b6d)"}}

Node logs output the following:

ERROR[09-20|00:08:54.573] Invalid receipt array RLP                hash=ff68a7..396b6d err="rlp: expected input list for []*types.Log, decoding into ([]*rawdb.receiptLogs)[0](rawdb.storedReceiptRLP).Logs"
WARN [09-20|00:08:54.573] Served eth_getLogs                       conn=127.0.0.1:37962 reqid=1 duration="333.35µs" err="failed to get logs for block #7950923 (0xff68a7..396b6d)"

It seems to be a similar issue to this: #25809

Steps to reproduce the behaviour

curl --location 'localhost:8545' --header 'Content-Type: application/json' --data '{
    "method": "eth_getLogs",
    "params": [
        {
            "blockHash": "0xff68a7848d2ca566f2f3770001bc97858a747595952b26b962f37a3fe9396b6d"
        }
    ],
    "id": 1,
    "jsonrpc": "2.0"
}'

[s1na]

Can you please try eth_getBlockReceipts(hash) and report the result?

[shammahalchemy]

Sure thing.

curl localhost:8545 --header 'Content-Type: application/json' --data '{
    "method": "eth_getBlockReceipts",
    "params": [
        {
            "blockHash": "0x2acd4a70f418203eb43b3731a4fe96927e1c109a6a8e2c3d8d21d6d9af8f66db"
        }
    ],
    "id": 1,
    "jsonrpc": "2.0"
}'
{"jsonrpc":"2.0","id":1,"error":{"code":-32000,"message":"receipts length mismatch: 3 vs 0"}}

[s1na]

It would be helpful to see what is exactly persisted. Since it is an old block it is stored in the freezer and not kvdb.

Even though we have a method debug_dbGet for reading values from the kvdb we don't have something for reading freezer values. Once we have this method we can make progress on debugging the issue.

[s1na]

Ah pardon turns out we do have a method for that. Can you please try calling this? you will have to enable the debug namespace (via --http.api "eth,debug"):

debug_dbAncient("receipts", 1022389)

This is the value I'm getting on my node:

0xf872e5a0ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb825208c0e5a0b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f82a410c0e5a03530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f52182f618c0

[MariusVanDerWijden]

I think the issue is that we might have at some point changed the way we store receipts in the database.
What was the first version that you ran this node on? Or how long have you been running this node?

[shammahalchemy]

These nodes have been running for quite a long time. I guess we'll need to do a resync?

[karalabe]

@shammahalchemy Could you please check this comment: #30470 (comment) ?

[fjl]

@shammahalchemy Please run this to retrieve the raw value:

curl localhost:8545 --header 'Content-Type: application/json' --data '{
    "method": "debug_dbAncient",
    "params": [ "receipts", 1022389 ],
    "id": 1,
    "jsonrpc": "2.0"
}'

[shammahalchemy]

@fjl sure thing. Thank you for looking.

{"jsonrpc":"2.0","id":1,"result":"0xf9042cf90161a0ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb825208b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0785afabffc4683bc75cd796dcd00468c451dbacdaf9abaff9d2ada054c6f7c2c940000000000000000000000000000000000000000c0825208f90161a0b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f82a410b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a0b3366c8a993f2bd20ebcc5c0337ae39cce2e06b92cd208480cccd32e5ea4ecd1940000000000000000000000000000000000000000c0825208f90161a03530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f52182f618b9010000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000a007a3187b343af8d2bdfcdbed2cbe095daeb43eeae2cc4d3172aa678e1d220fa5940000000000000000000000000000000000000000c0825208"}

[s1na]

rlpdump:

[
  [
    ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb,
    5208,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    785afabffc4683bc75cd796dcd00468c451dbacdaf9abaff9d2ada054c6f7c2c,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
  [
    b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f,
    a410,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    b3366c8a993f2bd20ebcc5c0337ae39cce2e06b92cd208480cccd32e5ea4ecd1,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
  [
    3530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f521,
    f618,
    00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000,
    07a3187b343af8d2bdfcdbed2cbe095daeb43eeae2cc4d3172aa678e1d220fa5,
    0000000000000000000000000000000000000000,
    [],
    5208,
  ],
]

Vs the correct dump:

[
  [
    ceabd1a5ecc5ddda016deb638f0609dd7ad3254b2ff4117cfeb062ff7b6e01fb,
    5208,
    [],
  ],
  [
    b5ff8dd2a923204fe1bc9c9875518f139acca756a79139cae6907d0582b40a7f,
    a410,
    [],
  ],
  [
    3530d12c12f5b965546f432d4fe9ec388ecd22e161bb84089e81e4f0a6b9f521,
    f618,
    [],
  ],
]

Yeah Marius was right this is a legacy receipt type. @shammahalchemy were you using --ignore-legacy-receipts? We even dropped that flag. I'm not sure how geth was starting for you 2 years ago when we were urging users to migrate from the legacy format (context: #26225). I am really puzzled how there is still a node with that format.

[fjl]

There is no conversion path in the code for legacy receipts anymore. So the only option is resync without ancients.