Snapshot tests

[onbjerg]

This issue is to discuss the possibility of using snapshot testing in various capacities.

Places I think it will definitely make sense to use snapshot tests:

• forge fmt tests since it takes some text and transforms it. Updating the tests for a full-fledged formatting run can end up being a pain.
• forge flatten tests. Same reasoning as above.

Additionally, it might make sense to use snapshot tests for a larger Solidity test suite. Currently we have testdata directories in multiple crates, some with duplicate tests, and some that are entirely unused. Some crates refer directly to evm-adapters/testdata instead.

I propose we create a top-level directory with various tests inside. Each test would be in its own folder and could either be a full project (for CLI integration tests) or single files (for behavioural tests).

There are also some obvious drawbacks to snapshot tests however that we must keep in mind:

• We need to be strict with accepting changes to snapshots
• We need to be careful not to overuse snapshot tests, i.e. we should only use snapshot tests in cases where we want to test large outputs against complex inputs. If we're testing against a very specific behaviour, e.g. that a field or two have a specific value, then we should use unit tests instead.

What are your thoughts?

Snapshot test library for Rust: https://crates.io/crates/insta

[onbjerg]

See also a thought I had for the behavioural tests for the test runners that might be better: https://github.com/onbjerg/foundry-test (the repo is a test suite I'm building up while migrating to REVM)

[gakonst]

Supportive of the idea, no strong opinions on the form factor