bug(`anvil`): memory / storage leak in long running test against Anvil #3478

kubmichael · 2022-10-12T06:06:12Z

Component

Anvil

Have you ensured that all of these are up to date?

Foundry
Foundryup

What version of Foundry are you on?

No response

What command(s) is the bug in?

No response

Operating System

Linux

Describe the bug

Hey, I run a long test with 10K blocks on anvil.
After some time I saw that memory and and storage usage of anvil has increased drastically.
I found that anvil is keeping mined blocks at file system as json files, also after digging in code, I think that blocks also kept in memory.
Is it intentional? Can I limit in some way number of blocks that kept in memory or at file system ?
Can I configure where to keep those files?

Thanks

mattsse · 2022-10-12T10:16:44Z

we keep the entire state in memory but cache history states on disk, however, there's still overhead.

there are currently no options to configure this, but this could be added very easily, including whether to keep the historic state at all, which should decrease memory footprint.

kubmichael · 2022-10-12T13:20:20Z

configuration option will be very helpful.

yhayun · 2022-10-12T14:02:00Z

+1 on adding an option to configure this option - the memory/storage footprint is significant

wminshew · 2022-10-12T22:30:16Z

I think this might be related but I've noticed the tests in https://github.com/0xSplits/splits-liquid slow down significantly after a few runs & eventually just start intermittently failing [FAIL. Reason: EvmError: FatalExternalError] testCan_distributeToMaxUniqueHolders() (gas: 9223372036854754743) (normal success case is ~110m gas?)

Arieg419 · 2022-10-13T13:56:46Z

+1 on adding this - this would be super impactful and significantly improve performance on long-running simulations.

gakonst · 2022-10-14T15:15:53Z

Can folks check if running tests with #3488 fixes it?

paco0x · 2022-10-17T04:50:12Z

Can folks check if running tests with #3488 fixes it?

Hi @gakonst, this issue still exists. I tested with the --prune-history flag, memory usage is still increasing with txs running. It consumes 12.7 G memory with about 55k txs executed on my end:

gakonst · 2022-10-17T14:04:49Z

@mattsse can u take a look here?

mattsse · 2022-10-17T14:09:19Z

@paco0x how can I reproduce this?

paco0x · 2022-10-17T16:37:26Z

@paco0x how can I reproduce this?

I'm running a simulation script that replays thousands of historical txs on Uniswap V3 in a forked network.

Sorry, the code cannot share here but I assume the issue is not related to any specified contract. All the APIs used in this script are:

anvil_setBalance
anvil_impersonateAccount
anvil_setStorageAt
anvil_setCode
evm_setNextBlockTimestamp
eth_call
eth_sendTransaction

With the transactions being executed in this forked network, the memory footprint of anvil also grows. And finally, the system got OOM.

As aforementioned, anvil is started with --prune-history and the memory usage is about 12.7G after 55k tx executed.

I also tried to call anvil_reset after the memory usage explosion. And the memory usage didn't reduce after the call, which is also unexpected.

Anvil is keeping all mined transaction in memory, so proposal is to add config option that will limit number of block with transaction in memory This is workaround Related to foundry-rs#3478

…#3836) * Due our work with anvil we faced huge memory consumption for long tests Anvil is keeping all mined transaction in memory, so proposal is to add config option that will limit number of block with transaction in memory This is workaround Related to #3478 * make clippy happy

0xalecks · 2023-01-04T16:04:05Z

We're also seeing memory issues while using a local subgraph trying to index a large set of events. Eventually Anvil stops responding.

gakonst · 2023-01-04T16:44:07Z

Would like to see a repro please, hard to debug otherwise! Thanks and appreciate the help/understanding.

Did you use the fix from #3836?

marshall-bean · 2023-01-06T13:34:13Z

Would like to see a repro please, hard to debug otherwise! Thanks and appreciate the help/understanding.

Did you use the fix from #3836?

I'm working with @0xalecks on this. Repro steps are here: https://gist.github.com/0xalecks/b42bc64302043b16cfe6e9464eadcb5f

I tried the fix in #3836 but did not notice any improvement. Thanks.

gakonst · 2023-01-06T13:34:56Z

Got it - will investigate, ty for the repro.

What paremeter did you set for transaction-block-keeper?

marshall-bean · 2023-01-06T13:36:12Z

Got it - will investigate, ty for the repro.

What paremeter did you set for transaction-block-keeper?

I tried --transaction-block-keeper 100 just as a SWAG. If there's something more appropriate I can try a different value.

0xalecks · 2023-01-18T16:13:47Z

Any updates on this? We're on hold waiting to see if this is something that might be fixed in Anvil or if we have to switch to a completely different solution. 🙏

gakonst · 2023-01-18T22:59:42Z

We'll take a pass this week - thanks! @mattsse PTAL, repro here

Was hoping somebody would've given some more debug information by now vs assuming we'll do all the debugging end to end^^

mattsse · 2023-01-19T11:03:07Z

@marshall-bean sorry for the wait.

I tried to replicate this but the add-dev-graph-node seems to be gone:

fatal: Remote branch add-dev-graph-node not found in upstream origin

does this also happen on master?

could you perhaps please also try with the --prune-history flag

marshall-bean · 2023-01-19T17:20:06Z

@marshall-bean sorry for the wait.

I tried to replicate this but the add-dev-graph-node seems to be gone:

fatal: Remote branch add-dev-graph-node not found in upstream origin

does this also happen on master?

could you perhaps please also try with the --prune-history flag

Yes is also happens on master, apologies. Please let me know if you have any further issues reproducing.

I can try again with --prune-history if you think that will help.

gakonst · 2023-01-19T20:00:55Z

Yeah pls give it a shot @marshall-bean

mattsse · 2023-01-19T20:29:36Z

I failed to get it working on first try, start timed out...

can try again in a bit

marshall-bean · 2023-01-20T16:06:47Z

Yeah pls give it a shot @marshall-bean

We tested this out yesterday and found that --prune-history seems to have solved the memory issue for us. We can consider this resolved from our POV, thank you.

mattsse · 2023-01-20T16:10:18Z

great, so this seems to be an issue with the default limits for historic state that we keep in memory.

I guess we can make this smarter and auto adjust best on some estimation

gakonst · 2023-01-20T18:36:22Z

I guess we could turn prune history and max transaction keeper on by default with some reasonable bounds?

… (#3836) * Due our work with anvil we faced huge memory consumption for long tests Anvil is keeping all mined transaction in memory, so proposal is to add config option that will limit number of block with transaction in memory This is workaround Related to foundry-rs/foundry#3478 * make clippy happy

wakamex · 2023-06-14T20:01:34Z

I guess we could turn prune history and max transaction keeper on by default with some reasonable bounds?

has this been turned on?

I wonder if it could be the source of this error:

ValueError: {'code': -32602, 'message': 'BlockOutOfRangeError: block height is 9331 but requested was 5700'}

grandizzy · 2024-09-26T06:09:00Z

this should be solved now with #8412, please retest and reopen if still an issue. thank you

kubmichael added the T-bug Type: bug label Oct 12, 2022

rkrasiuk added the C-anvil Command: anvil label Oct 12, 2022

mattsse mentioned this issue Oct 13, 2022

feat(anvil): add --prune-history flag #3488

Merged

mattsse mentioned this issue Oct 17, 2022

fix(anvil): clear db on reset #3505

Merged

kubmichael mentioned this issue Dec 5, 2022

Due our work with anvil we faced huge memory consumption for long t… #3836

Merged

gakonst assigned mattsse Jan 18, 2023

sabotagebeats mentioned this issue May 26, 2023

docs(anvil) --transaction-block-keeper flag is not documented #5055

Open

zerosnacks mentioned this issue Jun 27, 2024

meta(anvil): tracking issue for Anvil improvements #8269

Open

zerosnacks added this to the v1.0.0 milestone Jul 26, 2024

zerosnacks added the A-testing Area: testing label Aug 6, 2024

zerosnacks changed the title ~~Memory/storage leak in long test~~ bug(anvil): memory / storage leak in long running test against Anvil Aug 6, 2024

grandizzy closed this as completed Sep 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bug(`anvil`): memory / storage leak in long running test against Anvil #3478

bug(`anvil`): memory / storage leak in long running test against Anvil #3478

kubmichael commented Oct 12, 2022 •

edited

Loading

mattsse commented Oct 12, 2022

kubmichael commented Oct 12, 2022

yhayun commented Oct 12, 2022

wminshew commented Oct 12, 2022

Arieg419 commented Oct 13, 2022

gakonst commented Oct 14, 2022

paco0x commented Oct 17, 2022 •

edited

Loading

gakonst commented Oct 17, 2022

mattsse commented Oct 17, 2022

paco0x commented Oct 17, 2022 •

edited

Loading

0xalecks commented Jan 4, 2023

gakonst commented Jan 4, 2023 •

edited

Loading

marshall-bean commented Jan 6, 2023 •

edited

Loading

gakonst commented Jan 6, 2023

marshall-bean commented Jan 6, 2023

0xalecks commented Jan 18, 2023

gakonst commented Jan 18, 2023 •

edited

Loading

mattsse commented Jan 19, 2023 •

edited

Loading

marshall-bean commented Jan 19, 2023

gakonst commented Jan 19, 2023

mattsse commented Jan 19, 2023

marshall-bean commented Jan 20, 2023 •

edited

Loading

mattsse commented Jan 20, 2023

gakonst commented Jan 20, 2023

wakamex commented Jun 14, 2023

grandizzy commented Sep 26, 2024

bug(anvil): memory / storage leak in long running test against Anvil #3478

bug(anvil): memory / storage leak in long running test against Anvil #3478

Comments

kubmichael commented Oct 12, 2022 • edited Loading

Component

Have you ensured that all of these are up to date?

What version of Foundry are you on?

What command(s) is the bug in?

Operating System

Describe the bug

mattsse commented Oct 12, 2022

kubmichael commented Oct 12, 2022

yhayun commented Oct 12, 2022

wminshew commented Oct 12, 2022

Arieg419 commented Oct 13, 2022

gakonst commented Oct 14, 2022

paco0x commented Oct 17, 2022 • edited Loading

gakonst commented Oct 17, 2022

mattsse commented Oct 17, 2022

paco0x commented Oct 17, 2022 • edited Loading

0xalecks commented Jan 4, 2023

gakonst commented Jan 4, 2023 • edited Loading

marshall-bean commented Jan 6, 2023 • edited Loading

gakonst commented Jan 6, 2023

marshall-bean commented Jan 6, 2023

0xalecks commented Jan 18, 2023

gakonst commented Jan 18, 2023 • edited Loading

mattsse commented Jan 19, 2023 • edited Loading

marshall-bean commented Jan 19, 2023

gakonst commented Jan 19, 2023

mattsse commented Jan 19, 2023

marshall-bean commented Jan 20, 2023 • edited Loading

mattsse commented Jan 20, 2023

gakonst commented Jan 20, 2023

wakamex commented Jun 14, 2023

grandizzy commented Sep 26, 2024

bug(`anvil`): memory / storage leak in long running test against Anvil #3478

bug(`anvil`): memory / storage leak in long running test against Anvil #3478

kubmichael commented Oct 12, 2022 •

edited

Loading

paco0x commented Oct 17, 2022 •

edited

Loading

paco0x commented Oct 17, 2022 •

edited

Loading

gakonst commented Jan 4, 2023 •

edited

Loading

marshall-bean commented Jan 6, 2023 •

edited

Loading

gakonst commented Jan 18, 2023 •

edited

Loading

mattsse commented Jan 19, 2023 •

edited

Loading

marshall-bean commented Jan 20, 2023 •

edited

Loading