feat: eth: optimize eth block loading + eth_feeHistory #10446

Stebalien · 2023-03-11T00:08:57Z

Related Issues

The current implementations of eth_getBlock* and eth_feeHistory have proven computationally expensive due to their usage of full tipset computations (StateCompute). This PR replaces the expensive logic with receipt lookups.

Proposed Changes

For eth_getBlock* and eth_feeHistory, instead of replaying the tipset:

Lookup the tipset receipts.
Load the receipts.
Calculate everything from there (we have the gas used, etc.)

This also removes support for the pending block parameter as it's impossible to handle it without forcing state computes, which are prohibitively expensive. The Eth community is evaluating a proposal to also remove it in their next fork, given its limited applications due to MEV and other reasons.

Additional Info

We've gone down from 10 seconds to various minutes (depending on the method and amount of data request) to almost instantaneous responses.

After

⟩ time curl --silent --url http://localhost:1234/rpc/v1 \
      -X POST \
      -H "Content-Type: application/json" \
      -d '{"jsonrpc":"2.0","method":"eth_feeHistory","params":[ "0x08", "latest", [10, 50, 90] ],"id":1}'
{"jsonrpc":"2.0","result":{"oldestBlock":"0x28d39e","baseFeePerGas":["0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b","0x56ceb9b"],"gasUsedRatio":[1.1866695961,2.0196181532,2.5688713134,1.6794295302,1.7004244153,1.2444834388,1.736932316,2.2669179384],"reward":[["0x2de55df","0x2e29630","0x2e8e694"],["0x2d92b3b","0x2e665f8","0x5a874a8"],["0x294d17d","0x2e5547e","0x2eb2dbb"],["0x11c1e0d","0x2e35135","0x38a8b8f"],["0x11b8225","0x2e4b13e","0x39fb27c"],["0x2e18bb6","0x2e7c695","0x5d51c6a"],["0x2e0836d","0x2e70add","0x2ec2445"],["0x2e18763","0x2e429c9","0x2ebcc8a"]]},"id":1}

________________________________________________________
Executed in  345.14 millis    fish           external
   usr time    3.56 millis    0.12 millis    3.44 millis
   sys time    7.72 millis    3.36 millis    4.36 millis

~/W/pl/lotus · (steb/optimize-eth-block±)
⟩ time curl --silent --url http://localhost:1234/rpc/v1 \
      -X POST \
      -H "Content-Type: application/json" \
      -d '{"jsonrpc":"2.0","method":"eth_getBlockByNumber","params":[ "latest", false ],"id":1}'
...

________________________________________________________
Executed in  366.47 millis    fish           external
   usr time    4.64 millis    0.12 millis    4.52 millis
   sys time   11.66 millis    4.58 millis    7.08 millis

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title is in the form of of <PR type>: <area>: <change being made>
- example: fix: mempool: Introduce a cache for valid signatures
- PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
- area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state.

Read the receipts all at once instead of fetching them one-by-one.

Stebalien · 2023-03-11T00:11:21Z

Blocked on #10445.

Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways.

raulk

This looks good, about to test it.

itests/eth_block_hash_test.go

After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too.

raulk

I migrated eth_feeHistory to reading receipts, and dropped support for the pending block parameter. Fleshed out the description and added numbers. This is ready to go.

raulk · 2023-03-11T23:08:44Z

@Stebalien can you review my changes?

Stebalien

I can't approve but the new changes look correct.

node/impl/full/eth.go

Stebalien · 2023-03-11T23:28:02Z

node/impl/full/eth.go

 	for blocksIncluded < int(params.BlkCount) && ts.Height() > 0 {
-		compOutput, err := a.StateCompute(ctx, ts.Height(), nil, ts.Key())
+		msgs, rcpts, err := messagesAndReceipts(ctx, ts, a.Chain, a.StateAPI)


This works but it isn't great. Ideally we'd just walk the chain backwards as described in #10288.

Aren't the mechanics the same?

Stebalien · 2023-03-11T23:30:18Z

It looks like we need to fix the conformance tests to drop the "pending" one.

feat: eth: optimize eth block loading + eth_feeHistory

icodeface · 2023-03-15T08:43:35Z

How to get correct nonce with pending txs after "pending" param is dropped ?

go-lifei · 2023-03-15T08:48:48Z

How to get correct nonce with pending txs after "pending" param is dropped ?

    let nonce = await ethers.provider.getTransactionCount(
        addr,
        'latest'
    )

icodeface · 2023-03-15T08:55:18Z

getTransactionCount

This doesn't take into transactions in the mempool, which could cause nonce collisions.

Stebalien added 2 commits March 10, 2023 15:33

feat: eth: optimize receipt reading

1cf57ff

This optimizes the eth APIs (except the fee history one) to lookup the tipset state/receipts instead of computing the state.

feat: api: optimize ChainGetParentReceipts

eba270d

Read the receipts all at once instead of fetching them one-by-one.

Stebalien marked this pull request as ready for review March 11, 2023 00:11

Stebalien requested a review from a team as a code owner March 11, 2023 00:11

Stebalien marked this pull request as draft March 11, 2023 00:11

Stebalien and others added 2 commits March 10, 2023 16:29

test: eth: reduce chances of chain-reorgs affecting the test

59bebf8

Now that this API is _much_ faster, we're more likely to "catch up" to the head faster than it can stabilize. I'm pretty sure the test was intended to be written this way anyways.

Merge branch 'master' into steb/optimize-eth-block

d1c6ab7

raulk approved these changes Mar 11, 2023

View reviewed changes

itests/eth_block_hash_test.go Show resolved Hide resolved

raulk mentioned this pull request Mar 11, 2023

make StateCompute not write the output state to the blockstore. #10453

Closed

7 tasks

eth_feeHistory: migrate to using TipSetState.

f7a979d

raulk marked this pull request as ready for review March 11, 2023 22:28

Eth API: drop support for 'pending' block parameter.

9412753

After transitioning from using StateCompute to loading receipts, we can no longer handle the 'pending' block without forcing computation. Eth Core Devs are evaluating a proposal to remove support on their end too.

raulk approved these changes Mar 11, 2023

View reviewed changes

Stebalien commented Mar 11, 2023

View reviewed changes

raulk mentioned this pull request Mar 12, 2023

feat: discard state writes in StateCompute and StateReplay. #10457

Closed

7 tasks

raulk added 2 commits March 12, 2023 00:44

remove superfluous filter.

5a4b5ff

drop irrelevant test.

571a84b

raulk added the release/backport label Mar 12, 2023

raulk changed the title ~~feat: eth: optimize eth block loading~~ feat: eth: optimize eth block loading + eth_feeHistory Mar 12, 2023

raulk enabled auto-merge March 12, 2023 00:58

raulk added this to the Eth RPC perf + fixes milestone Mar 12, 2023

raulk disabled auto-merge March 12, 2023 15:45

raulk merged commit aea195a into master Mar 12, 2023

raulk deleted the steb/optimize-eth-block branch March 12, 2023 15:46

Fatman13 mentioned this pull request Mar 13, 2023

[venus] eth optimization / eth 优化合集 filecoin-project/venus#5812

Closed

19 tasks

raulk added a commit that referenced this pull request Mar 13, 2023

Merge pull request #10446 from filecoin-project/steb/optimize-eth-block

d4dc28c

feat: eth: optimize eth block loading + eth_feeHistory

raulk added a commit that referenced this pull request Mar 13, 2023

Merge pull request #10446 from filecoin-project/steb/optimize-eth-block

d4d06e6

feat: eth: optimize eth block loading + eth_feeHistory

Terryhung mentioned this pull request Mar 14, 2023

feat:update the lotus version filecoin-project/lily#1153

Merged

raulk modified the milestones: Eth RPC perf + fixes, v1.20.4 Mar 15, 2023

jennijuju modified the milestones: v1.20.4, v1.21.0 Mar 16, 2023

Stebalien removed the release/backport label Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: eth: optimize eth block loading + eth_feeHistory #10446

feat: eth: optimize eth block loading + eth_feeHistory #10446

Stebalien commented Mar 11, 2023 •

edited by raulk

Loading

Stebalien commented Mar 11, 2023

raulk left a comment

raulk left a comment

raulk commented Mar 11, 2023

Stebalien left a comment

Stebalien Mar 11, 2023

raulk Mar 12, 2023

Stebalien commented Mar 11, 2023

icodeface commented Mar 15, 2023

go-lifei commented Mar 15, 2023

icodeface commented Mar 15, 2023

feat: eth: optimize eth block loading + eth_feeHistory #10446

feat: eth: optimize eth block loading + eth_feeHistory #10446

Conversation

Stebalien commented Mar 11, 2023 • edited by raulk Loading

Related Issues

Proposed Changes

Additional Info

After

Checklist

Stebalien commented Mar 11, 2023

raulk left a comment

Choose a reason for hiding this comment

raulk left a comment

Choose a reason for hiding this comment

raulk commented Mar 11, 2023

Stebalien left a comment

Choose a reason for hiding this comment

Stebalien Mar 11, 2023

Choose a reason for hiding this comment

raulk Mar 12, 2023

Choose a reason for hiding this comment

Stebalien commented Mar 11, 2023

icodeface commented Mar 15, 2023

go-lifei commented Mar 15, 2023

icodeface commented Mar 15, 2023

Stebalien commented Mar 11, 2023 •

edited by raulk

Loading