feat(anvil): support state caching for forked historical blocks

[asafyish]

Component

Anvil

Describe the feature you would like

I am running anvil in fork mode (connected to archive node), like so:

anvil --fork-url "https://eth-mainnet.alchemyapi.io/v2/XXXXXXXX" --fork-block-number 14577209

Then I am trying to read historical blocks data, which works, but only caches block 14577209, even though I am reading from the other blocks. The same scenario does create cache in hardhat and ganache. Since anvil is so much faster in the initial reading, I prefer using it.

This is how I am reading historical data:

const ethers = require("ethers");

const daiAddress = "0x6B175474E89094C44Da98b954EedeAC495271d0F";
const daiAbi = ["function name() view returns (string)"];

const provider = new ethers.providers.JsonRpcProvider("http://127.0.0.1:8545");
const daiContract = new ethers.Contract(daiAddress, daiAbi, provider);

for (let blockTag = 14577100; blockTag < 14577209; blockTag++) {
  const name = await daiContract.name({
    blockTag,
  });
  console.log(name);
}

Running

forge cache ls

returns:
-️ mainnet (2.1 kB)
-️ Block 14577209 (2.1 kB)

Additional context

forge 0.2.0 (6ca977f 2022-05-29T02:10:50.584077Z)

[mattsse]

currently we only store the state of the forked block, and cache is blockwise.

if the block predates the fork we simply delegate the eth_call.
in order to support state caching for these blocks we'd need to execute these calls locally and instead let the evm translate this to storage requests so we can cache single storage slots.

which would be doable but would require some work.

I'm not sure how hardhat/ganache store that eth_call, are they storing state map or eth_call/response pairs?

[asafyish]

I am not familiar with hardhat implementation, but looking into the cache folder, it seems they are creating a "request-{transaction-hash}.json" file, with the contents of the response inside. I don't see any block number reference, so I can only guess that part of the hash contains the block number, something like sha(blockNumber + transaction-data).

[mds1]

I looked into hardhat's caching approach a little while back (see #698 (comment)). The answer is yes, they store call/response pairs:

• cache/hardhat-network-fork/network-{chainId}/ is the path containing cached responses for a given chain ID
• Within that folder, each request is stored as a separate JSON file called request-{key}.json, where the key is a hash of {chain ID} {rpcMethod} {stringifiedParams}. You can see the implementation here.
• The contents of each JSON file is simply the response of the request
• They also don't cache blocks that are too recent and at risk of being reorged, and that threshold varies by network

Most (all?) of their implementation can be found here: https://github.com/nomiclabs/hardhat/blob/4f108b51fc7f87bcf7f173a4301b5973918b4903/packages/hardhat-core/src/internal/hardhat-network/jsonrpc/client.ts#L224-L475

[tynes]

Needing to implement chain specific logic for safe depth could be implemented in a more generic way if the corresponding blockhash is stored locally and always checked against the remote before using the cache. If they don't match, then flush the cache. That adds an extra http request but it can be amortized over many possible state lookups

[mds1]

Ah yes good point, did not mean to imply we should use hardhat's approach of a chain-specific reorg threshold—I agree the blockhash approach is better and forgot to mention it

[onbjerg]

Is this still the case? If so, is this expected/is it fixable? cc @mattsse

[julien-devatom]

which would be doable but would require some work.

Is it still considered to have a call cache that can be dumped into a file?
My point is when you are running 1M times the same calls every day by using anvil. You can only leverage the in-memory caching of the storage. So when restarting anvil, you're restarting from scratch.

That would be an awesome feature.

[grandizzy]

we now have

      --dump-state <PATH>
          Dump the state and block environment of chain on exit to the given file.
          If the value is a directory, the state will be written to `<VALUE>/state.json`.
      --load-state <PATH>
          Initialize the chain from a previously saved state snapshot

flags which could be used (we're fixing the issue reported in #8493 though)

Would these new options work for mentioned scenario? thank you

CC @julien-devatom @asafyish

[Maliksb11]

OK

[grandizzy]

ping @julien-devatom @asafyish could you pls check comment above? thanks!