meta: invariant testing features

[mds1]

Component

Forge

Describe the feature you would like

This issue simply aggregates the issues that would improve invariant testing (and some related to fuzz testing too). In no particular order:

• 1. feat (proposal): UX for granularity over fuzz, invariant, and solidity sections #3062
• 2. feat: commit state changes to test contract during invariant/fuzz tests #3005
• 3. feat: weight invariant selectors by number of selectors, not number of contracts. #2986
• 4. bug: invariant test chooses contract senders which are rejected due to EIP-3607 #2963
• 5. Ability to not allow random callers in invariant testing #3400
• 6. feat: vm.depth() cheatcode to return the depth of the current invariant run #2985
• 7. bug: invariant test log/traces not shown #2962
• 8. feat: fuzz failure persistence #2551
• 9. feat: fuzz corpus saving and replay in standard format #2552
• 10. feature: assumeRange(uint lower, uint upper) in Vm.sol #1350
• 11. feat: test for reentrancy in invariant tests #1578
• 12. feat(cheatcodes): ability to capture and store state diffs #2846
• 13. meta(tracing): tracking issue for tracing improvements #3390
• 14. Progress on Invariant Tests #3277 (listing this as a generic issue for implementing additional invariant techniques/strategies)
• 15. perf: fuzz/invariant benchmarks #3411
• 16. feat: more flexible/powerful ways to define and test invariants #3452
• 17. feat: deduplicate fuzz inputs #3521
• 18. feat: add real-time counter for fuzz/invariant runs #585
• 19. feat(forge): Add internal metrics capability #3607
• 20. feat: vm.assume support in invariant tests #4190
• 21. Add invariant testing filter for excludeSelectors() #4352
• 22. feat: Add "after-all" hook for testing #4300
• 23. feat(forge): exclude precompiles by default in invariant tests #4287
• 24. bug(forge): custom errors are not decoded in invariant test traces #4178
• 25. Built-in contracts like vm and the create2 factory should be excluded senders in invariants #4163
• 26. Other ways to improve fuzz/invariant tests, like coverage guidance or things mentioned in Progress on Invariant Tests #3277

Additional context

No response

[lucas-manuel]

Adding metrics issue as per our discussion @mds1 @gakonst

#3607

[gakonst]

Caught up with @LucasManuel @mds1. Let's prioritize: #3005 #2963 #2985 and #2962.

[mds1]

Two last requests, lower priority than the above set but still valuable + should be quick ones:

• feat: weight invariant selectors by number of selectors, not number of contracts. #2986 would enable better support for multiple, different actors + it's better for the standard usage as well
• feat: add real-time counter for fuzz/invariant runs #585 to better support long-running campaigns (to have confidence it's not hanging)

[mds1]

Closing this in favor of #4438