Lack of transaction receipts (eth_getTransactionReceipt) for some of FEVM transactions

[MarcinZakrzewski13]

Checklist

• This is not a security-related bug/issue. If it is, please follow please follow the security policy.
• I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
• I did not make any code changes to lotus.

Lotus component

• lotus Ethereum RPC
• lotus FVM - Lotus FVM interactions
• FEVM tooling
• Other

Lotus Version

Daemon:  1.23.3+mainnet+git.7bb1f98ac+api1.5.0
Local: lotus version 1.23.3+mainnet+git.7bb1f98ac

Repro Steps

Request

curl --request POST --url https://node-URI --header 'Content-Type: application/json' --header 'X-API-Key: api-key' --data '{"jsonrpc": "2.0","method": "eth_getTransactionReceipt","params": ["0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325"],"id": 1}' 

Response

{"jsonrpc":"2.0","result":null,"id":1}

Describe the Bug

We are indexing Filecoin blocks using Lotus node. We have found that for transactions in FEVM we did not get transaction receipts.

Today, we have indexed data from blocks in range from 2_683_348 to 2_846_394 searching of transactions with missing receipt.
In this range we found 53_082 transactions without receipt in 11_068 blocks.

First block with transaction which has not receipt on node: 2_683_360
Last block with transaction which has not receipt on node: 2_792_027

Is it a node issue or something we don't see?

Samples

Transactions which have not receipt on node:

2683360,0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325,bafy2bzacebctgcd4fi5m23446hzhuel7mbxnotffkkufuqgpcwgbisduy3ujk
Explorer: https://filfox.info/en/message/0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325?t=3
Node:

{"jsonrpc":"2.0","result":null,"id":1}

2683375,0x67e59d73c92fa1c0cfc666038fc52a7264135d9d5b66434a77fcef548c383113,bafy2bzaceblwl7zvemx5ovsxhryjdjb4npnlyilmwtktj77qp22ni2atccqvo
Explorer: https://filfox.info/en/message/0x67e59d73c92fa1c0cfc666038fc52a7264135d9d5b66434a77fcef548c383113?t=3
Node:

{"jsonrpc":"2.0","result":null,"id":1}

2683381,0x03cf7eaf8a370974988affde7ee4527c379c3cb4b60af552064995e6a4890d05,bafy2bzacec7247sycph26tnoaqnvo62no4pmtkosmedbnl2d3m5ucdzsnk22u
Explorer: https://filfox.info/en/message/0x03cf7eaf8a370974988affde7ee4527c379c3cb4b60af552064995e6a4890d05?t=3
Node:

{"jsonrpc":"2.0","result":null,"id":1}

Transactions which have receipt on node

2683363,0x3a59331d838a4c0da768bc0ec02904aa6e19f9fd6748ce60c1bec649919a4573,bafy2bzaceamyzkhzoe4v3esemhyzuaqv2tllne3dyf7kwsrnmwvh77y7kunri
Explorer: https://filfox.info/en/message/0x3a59331d838a4c0da768bc0ec02904aa6e19f9fd6748ce60c1bec649919a4573?t=3
Node:

 {"jsonrpc":"2.0","result": {"transactionHash":"0x3a59331d838a4c0da768bc0ec02904aa6e19f9fd6748ce60c1bec649919a4573","transactionIndex":"0x7b","blockHash":"0x2f4432275dc3342ffec1367e968550fe055d41bf68546a4be3de3bc00488c34f","blockNumber":"0x28f1e3","from":"0x6b9cd26e5238b1eb528db0f78be86c8804cb0af7","to":"0xa768b57c4186ad4a8715d6cecaca9eb04f3d734f","root":"0x0000000000000000000000000000000000000000000000000000000000000000","status":"0x1","contractAddress":null,"cumulativeGasUsed":"0x0","gasUsed":"0xc0c1e0","effectiveGasPrice":"0xba75d6d6","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","logs":[{"address":"0xa768b57c4186ad4a8715d6cecaca9eb04f3d734f","data":"0x","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000a768b57c4186ad4a8715d6cecaca9eb04f3d734f","0x0000000000000000000000000000000000000000000000000000000000000000"],"removed":false,"logIndex":"0x0","transactionIndex":"0x7b","transactionHash":"0x3a59331d838a4c0da768bc0ec02904aa6e19f9fd6748ce60c1bec649919a4573","blockHash":"0x2f4432275dc3342ffec1367e968550fe055d41bf68546a4be3de3bc00488c34f","blockNumber":"0x28f1e3"},{"address":"0xa768b57c4186ad4a8715d6cecaca9eb04f3d734f","data":"0x","topics":["0xe9274a84b19e9428826de6bae8c48329354f8f0e73f771b97cae2d9dccd45a27","0x000000000000000000000000a768b57c4186ad4a8715d6cecaca9eb04f3d734f","0x0000000000000000000000000000000000000000000000000000000000000000"],"removed":false,"logIndex":"0x1","transactionIndex":"0x7b","transactionHash":"0x3a59331d838a4c0da768bc0ec02904aa6e19f9fd6748ce60c1bec649919a4573","blockHash":"0x2f4432275dc3342ffec1367e968550fe055d41bf68546a4be3de3bc00488c34f","blockNumber":"0x28f1e3"}],"type":"0x2"},"id":1}

2683365,0xaaae15af843cda1981f33fea3d4b7dfd97abef277d94ceaf16a49af360d99830,bafy2bzacedzxyboeysnc552mboimvqraeipeqeibmgrkldfgpvp3ldzuzvz56
Explorer: https://filfox.info/en/message/0xaaae15af843cda1981f33fea3d4b7dfd97abef277d94ceaf16a49af360d99830?t=3
Node:

{"jsonrpc":"2.0","result":{"transactionHash":"0xaaae15af843cda1981f33fea3d4b7dfd97abef277d94ceaf16a49af360d99830","transactionIndex":"0x33","blockHash":"0xa6ac3496fd3ae3ba45a39d71225cbce72dd18f34674012a9309ea6be226159d8","blockNumber":"0x28f1e5","from":"0xf6adbfe2b3051a463afad071a54278869a084304","to":null,"root":"0x0000000000000000000000000000000000000000000000000000000000000000","status":"0x1","contractAddress":"0x7b90337f65faa2b2b8ed583ba1ba6eb0c9d7ea44","cumulativeGasUsed":"0x0","gasUsed":"0x69e24f1","effectiveGasPrice":"0xba9cd520","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","logs":[{"address":"0x7b90337f65faa2b2b8ed583ba1ba6eb0c9d7ea44","data":"0x0000000000000000000000000000000000000002863c1f5cdae42f9540000000","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x0000000000000000000000000000000000000000000000000000000000000000","0x00000000000000000000000086c215ba617fec5605b65e3bee6d86011bc80b80"],"removed":false,"logIndex":"0x0","transactionIndex":"0x33","transactionHash":"0xaaae15af843cda1981f33fea3d4b7dfd97abef277d94ceaf16a49af360d99830","blockHash":"0xa6ac3496fd3ae3ba45a39d71225cbce72dd18f34674012a9309ea6be226159d8","blockNumber":"0x28f1e5"},{"address":"0x7b90337f65faa2b2b8ed583ba1ba6eb0c9d7ea44","data":"0x0000000000000000000000000000000000000001431e0fae6d7217caa0000000","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000c549cf81554b2d6b9fc85edd5a1b004440742f75"],"removed":false,"logIndex":"0x1","transactionIndex":"0x33","transactionHash":"0xaaae15af843cda1981f33fea3d4b7dfd97abef277d94ceaf16a49af360d99830","blockHash":"0xa6ac3496fd3ae3ba45a39d71225cbce72dd18f34674012a9309ea6be226159d8","blockNumber":"0x28f1e5"}],"type":"0x2"},"id":1}

Tooling

FEVM JSON RPC API

Configuration Options

[Fevm]
# EnableEthRPC enables eth_ rpc, and enables storing a mapping of eth transaction hashes to filecoin message Cids.
# This will also enable the RealTimeFilterAPI and HistoricFilterAPI by default, but they can be disabled by config options above.
#
# type: bool
# env var: LOTUS_FEVM_ENABLEETHRPC
EnableEthRPC = true

# EthTxHashMappingLifetimeDays the transaction hash lookup database will delete mappings that have been stored for more than x days
# Set to 0 to keep all mappings
#
# type: int
# env var: LOTUS_FEVM_ETHTXHASHMAPPINGLIFETIMEDAYS
EthTxHashMappingLifetimeDays = 0

[Fevm.Events]
# EnableEthRPC enables APIs that
# DisableRealTimeFilterAPI will disable the RealTimeFilterAPI that can create and query filters for actor events as they are emitted.
# The API is enabled when EnableEthRPC is true, but can be disabled selectively with this flag.
#
# type: bool
# env var: LOTUS_FEVM_EVENTS_DISABLEREALTIMEFILTERAPI
DisableRealTimeFilterAPI = false

# DisableHistoricFilterAPI will disable the HistoricFilterAPI that can create and query filters for actor events
# that occurred in the past. HistoricFilterAPI maintains a queryable index of events.
# The API is enabled when EnableEthRPC is true, but can be disabled selectively with this flag.
#
# type: bool
# env var: LOTUS_FEVM_EVENTS_DISABLEHISTORICFILTERAPI
DisableHistoricFilterAPI = false

    # FilterTTL specifies the time to live for actor event filters. Filters that haven't been accessed longer than
    # this time become eligible for automatic deletion.
    #
    # type: Duration
    # env var: LOTUS_FEVM_EVENTS_FILTERTTL
    #FilterTTL = "24h0m0s"

    # MaxFilters specifies the maximum number of filters that may exist at any one time.
    #
    # type: int
    # env var: LOTUS_FEVM_EVENTS_MAXFILTERS
    #MaxFilters = 100

    # MaxFilterResults specifies the maximum number of results that can be accumulated by an actor event filter.
    #
    # type: int
    # env var: LOTUS_FEVM_EVENTS_MAXFILTERRESULTS
    #MaxFilterResults = 10000

    # MaxFilterHeightRange specifies the maximum range of heights that can be used in a filter (to avoid querying
    # the entire chain)
    #
    # type: uint64
    # env var: LOTUS_FEVM_EVENTS_MAXFILTERHEIGHTRANGE
    #MaxFilterHeightRange = 2880

    # DatabasePath is the full path to a sqlite database that will be used to index actor events to
    # support the historic filter APIs. If the database does not exist it will be created. The directory containing
    # the database must already exist and be writeable. If a relative path is provided here, sqlite treats it as
    # relative to the CWD (current working directory).
    #
# type: string
# env var: LOTUS_FEVM_EVENTS_DATABASEPATH
DatabasePath = "/data/node/actor_events/"

[i-norden]

FWIW our archive appears to have, at least some of, the missing tx receipts

Query:

iannorden@Ians-MacBook-Pro go-datastore % curl --request POST --url 'ourPrivateArchiveURL' --header 'Content-Type: application/json' --header 'X-API-Key: api-key' --data '{"jsonrpc": "2.0","method": "eth_getTransactionReceipt","params": ["0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325"],"id": 1}' 

Response:

{"jsonrpc":"2.0","result":{"transactionHash":"0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325","transactionIndex":"0x3b","blockHash":"0xf95216536f34d7cb3d93b8b80d66ae275f4b6db2274778ea2fc3710e87a50c4a","blockNumber":"0x28f1e0","from":"0x6b9cd26e5238b1eb528db0f78be86c8804cb0af7","to":null,"root":"0x0000000000000000000000000000000000000000000000000000000000000000","status":"0x1","contractAddress":"0xa768b57c4186ad4a8715d6cecaca9eb04f3d734f","cumulativeGasUsed":"0x0","gasUsed":"0x7810670","effectiveGasPrice":"0xba8d42e4","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","logs":[{"address":"0xa768b57c4186ad4a8715d6cecaca9eb04f3d734f","data":"0x","topics":["0x8be0079c531659141344cd1fd0a4f28419497f9722a3daafe3b4186f6b6457e0","0x0000000000000000000000000000000000000000000000000000000000000000","0x0000000000000000000000006b9cd26e5238b1eb528db0f78be86c8804cb0af7"],"removed":false,"logIndex":"0x0","transactionIndex":"0x3b","transactionHash":"0x197bb7dc4af0fa587b913827f6c626d695a5205427656e26f7637fba03f88325","blockHash":"0xf95216536f34d7cb3d93b8b80d66ae275f4b6db2274778ea2fc3710e87a50c4a","blockNumber":"0x28f1e0"}],"type":"0x2"},"id":1}

Query:

iannorden@Ians-MacBook-Pro go-datastore % curl --request POST --url 'ourPrivateArchiveURL' --header 'Content-Type: application/json' --header 'X-API-Key: api-key' --data '{"jsonrpc": "2.0","method": "eth_getTransactionReceipt","params": ["0x67e59d73c92fa1c0cfc666038fc52a7264135d9d5b66434a77fcef548c383113"],"id": 1}'

Response:
Too long to embed here (possibly why filfox throws a 404 for this one?)... response is here: https://gist.github.com/i-norden/325560c772c05ff5bc814d49b8b430c4

Query:

iannorden@Ians-MacBook-Pro go-datastore % curl --request POST --url 'ourPrivateArchiveURL' --header 'Content-Type: application/json' --header 'X-API-Key: api-key' --data '{"jsonrpc": "2.0","method": "eth_getTransactionReceipt","params": ["0x03cf7eaf8a370974988affde7ee4527c379c3cb4b60af552064995e6a4890d05"],"id": 1}' 

Response:

{"jsonrpc":"2.0","result":{"transactionHash":"0x03cf7eaf8a370974988affde7ee4527c379c3cb4b60af552064995e6a4890d05","transactionIndex":"0x68","blockHash":"0x987b59b6d71d154746b595b1fbe8bc263a76462a6346c6330a565e4cd2424eb5","blockNumber":"0x28f1f5","from":"0xf6adbfe2b3051a463afad071a54278869a084304","to":"0x7b90337f65faa2b2b8ed583ba1ba6eb0c9d7ea44","root":"0x0000000000000000000000000000000000000000000000000000000000000000","status":"0x1","contractAddress":null,"cumulativeGasUsed":"0x0","gasUsed":"0xbf7945","effectiveGasPrice":"0x1184472bc","logsBloom":"0xffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff","logs":[{"address":"0x7b90337f65faa2b2b8ed583ba1ba6eb0c9d7ea44","data":"0x00000000000000000000000000000000000000000004350e7d8114c452c00000","topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef","0x0000000000000000000000000000000000000000000000000000000000000000","0x000000000000000000000000f6adbfe2b3051a463afad071a54278869a084304"],"removed":false,"logIndex":"0x0","transactionIndex":"0x68","transactionHash":"0x03cf7eaf8a370974988affde7ee4527c379c3cb4b60af552064995e6a4890d05","blockHash":"0x987b59b6d71d154746b595b1fbe8bc263a76462a6346c6330a565e4cd2424eb5","blockNumber":"0x28f1f5"}],"type":"0x2"},"id":1}

[Stebalien]

My first concern is that, while we do backfill the index if the "txhash" database doesn't exist, that logic may not be race-free. However, the range is so large this seems unlikely.

Did you enable the Eth JSON-RPC API after syncing that range? In that case, you could be missing the events (which will, apparently, cause the lotus API to return "nothing" instead of an error in this case).

[MarcinZakrzewski13]

My first concern is that, while we do backfill the index if the "txhash" database doesn't exist, that logic may not be race-free. However, the range is so large this seems unlikely.

Did you enable the Eth JSON-RPC API after syncing that range? In that case, you could be missing the events (which will, apparently, cause the lotus API to return "nothing" instead of an error in this case).

We were using node set up by other company. This is the answer I've got from them

What we did was backfill from scratch.. but this is an archival node.

Also I'd like to add that after initial indexing process when we have found missing receipts we've made a check to investigate the scale of problem and after this check we have found that some of receipts were available - it seems that their availability has changed over time, so maybe they were indexed by the node until the code that was supposed to investigate the scale of the problem was run.

[Stebalien]

So, I think the issue is:

Did you enable the Eth JSON-RPC API after syncing that range? In that case, you could be missing the events (which will, apparently, cause the lotus API to return "nothing" instead of an error in this case).

Those events would be re-created if/when you re-execute a tipset (e.g., ask for an execution trace) which would explain why they're re-appearing. The solution would be to re-execute all of those tipsets (possibly in parallel).

If your upstream provider is willing to apply a patch, #11329 will give us actual error messages instead of just returning "nil" for missing receipts.

[Stebalien]

So, there's actually a decent solution: re-execute tipsets on-demand to fill in missing event information. We usually like to avoid this kind of thing for performance reasons however:

1. This will only apply to tipsets processed before enabling the Eth JSON-RPC API.
2. This will only happen once.

[MarcinZakrzewski13]

At the moment we've received access to other Lotus archive instance without described issues within task. The node instance with missing receipts errors was probably discarded.
From my point of view task can be closed. Decision on your side @i-norden @Stebalien .

[Stebalien]

I'd like to keep this open for now as it's still an issue and still something I'd like to eventually address.