bug(forge coverage): random setUp failed errors

[bearpebble]

Component

Forge

Have you ensured that all of these are up to date?

• Foundry
• Foundryup

What version of Foundry are you on?

forge 0.2.0 (89f74f2 2023-11-23T00:16:20.422885370Z)

What command(s) is the bug in?

forge coverage

Operating System

Linux

Describe the bug

Running forge coverage for the optimism tests sometimes leads to missing random parts of coverage.
Specifically, it would print setUp failed reason=execution error contract=0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496 whenever some coverage was missing.
The forge test command never exhibited that issue, so I'm assuming only the coverage command is affected.

The command never really fails, it just has missing coverage due to the above mentioned error while running the setUp function.
I finally managed to get it to fail while using RUST_LOG=trace and found that all of the failures had a similar pattern.

TestA: evm::executors: cleared broadcastable transactions
TestB: something with cheat codes
TestA: setUp failed

One actual trace snippet (there are several this run)

2023-11-27T14:06:35.579527Z DEBUG run_tests{name=contracts/test/Proxy.t.sol:Proxy_Test}: evm::executors: cleared broadcastable transactions
2023-11-27T14:06:35.579525Z TRACE run_tests{name=contracts/test/Hashing.t.sol:Hashing_hashDepositTransaction_Test}:fuzz-test{name=testDiff_hashDepositTransaction_succeeds(address,address,uint256,uint256,uint64,bytes,uint64) should_fail=false}:apply{cheat=toString_4Call { value: 0x000000000000000000000000000000000000000000000000000000a8157c9a9c_U256 }}: cheatcodes: applying
2023-11-27T14:06:35.579536Z TRACE run_tests{name=contracts/test/Hashing.t.sol:Hashing_hashDepositTransaction_Test}:fuzz-test{name=testDiff_hashDepositTransaction_succeeds(address,address,uint256,uint256,uint64,bytes,uint64) should_fail=false}:apply{cheat=toString_4Call { value: 0x000000000000000000000000000000000000000000000000000000a8157c9a9c_U256 }}: cheatcodes: return="0000000000000000000000000000000000000000000000000000000000000020000000000000000000000000000000000000000000000000000000000000000c3732313931343939333330380000000000000000000000000000000000000000"
2023-11-27T14:06:35.579546Z ERROR run_tests{name=contracts/test/Proxy.t.sol:Proxy_Test}: forge::runner: setUp failed reason=execution error contract=0x7FA9385bE102ac3EAc297483Dd6233D62b3e1496

The full trace can be found here:
https://drive.google.com/file/d/1VefltIFYAH1GULCmOkt7S5TWch3nToj_/view?usp=sharing

Because of this pattern and the execution only failing sometimes, I'm assuming it's a race condition.
Reproducing the issue is unfortunately not very consistent and it seems to fail more frequently on CI tests compared to local execution. I hope the trace is enough to find the root cause.

[bearpebble]

I believe this got fixed by #6451 since I haven't been able to reproduce the issue since
@mattsse if that sound reasonable we can close this issue

[0xfuturistic]

Ended up running into a similar issue with CI executing forge coverage --report lcov in Optimism's repo. Here's the issue in the other repo: ethereum-optimism/optimism#8705

[zerosnacks]

@bearpebble I'm attempting to reproduce this issue but running into stack too deep errors when running forge coverage --match-path test/L1/L2OutputOracle.t.sol in an effort to reproduce the issue, --via-ir does not help here.

@0xfuturistic I've been able to run the test suite without issue: ethereum-optimism/optimism#8705 (comment) but am unable to run forge coverage --report lcov with Foundry version: 63fff3510408b552f11efb8196f48cfe6c1da664 (as stated in versions.json) with solc 0.8.26 due to a stack too deep compilation error

I'll leave the issue open for now but would greatly appreciate a minimal reproduction if you are able to narrow it down or further instructions on running forge coverage --report lcov in the optimism repo

[grandizzy]

@zerosnacks I think we can close this one?
per @bearpebble

I believe this got fixed by #6451 since I haven't been able to reproduce the issue since
if that sound reasonable we can close this issue

as for the other issue reported, coverage disables any optimization which results in code not being able to be compiled, see ethereum-optimism/optimism#8705 (comment)

[zerosnacks]

Marking as resolved, feel free to re-open or leave a comment if you are still experiencing issues with this