Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fuzz): ability to declare fuzz test fixtures #7428

Merged
merged 31 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ef15bab
fix(fuzz): deduplicate fuzz inputs
grandizzy Mar 14, 2024
3c48666
Fix tests, collect fixtures in test setup, arc fixtures
grandizzy Mar 19, 2024
e1f8a95
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 19, 2024
254017c
Cleanup
grandizzy Mar 19, 2024
8586bf4
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 20, 2024
14c869a
Use fixture_ prefix
grandizzy Mar 20, 2024
d0442de
Update tests to reflect that random values are used if no fixtures
grandizzy Mar 20, 2024
7e4b14c
Review changes
grandizzy Mar 21, 2024
972d993
Group fuzz_calldata and fuzz_calldata_from_state in calldata mod
grandizzy Mar 21, 2024
708e4db
Review changes: remove unnecessary clones, nicer code to collect fixt…
grandizzy Mar 21, 2024
f93254a
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 22, 2024
9276893
Add support for bytes and string fixtures, fixture strategy macro. So…
grandizzy Mar 24, 2024
a5e8da0
Remove unnecessary clone
grandizzy Mar 26, 2024
14273fa
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 26, 2024
09ee66e
Use inline config
grandizzy Mar 26, 2024
342eaf7
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 26, 2024
4785f93
More robust invariant assume test
grandizzy Mar 26, 2024
0b7a230
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 5, 2024
f9adb66
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 10, 2024
fb86084
Fixtures as storage arrays, remove inline config
grandizzy Apr 10, 2024
6874ead
Simplify code
grandizzy Apr 10, 2024
ca4d44b
Support fixed size arrays fixtures
grandizzy Apr 10, 2024
acdf92d
Update comment
grandizzy Apr 11, 2024
12115fa
Use DynSolValue::type_strategy for address and fixed bytes fuzzed params
grandizzy Apr 11, 2024
e76fe8e
Add prefix to mark a storage array or a function as fixture
grandizzy Apr 11, 2024
3ac9e33
Fix test
grandizzy Apr 11, 2024
7e95208
Simplify code / fixture strategy macro, panic if configured fixture n…
grandizzy Apr 11, 2024
4cf19cb
Consistent panic with fixture strategy if uint / int fixture of diffe…
grandizzy Apr 13, 2024
53c64d3
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 17, 2024
997c5d4
Review changes: don't panic when invalid fixture, use prop_filter_map…
grandizzy Apr 18, 2024
a127780
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions crates/common/src/traits.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ pub trait TestFunctionExt {

/// Returns whether this function is a `setUp` function.
fn is_setup(&self) -> bool;

/// Returns whether this function is a fixture function.
fn is_fixture(&self) -> bool;
}

impl TestFunctionExt for Function {
Expand All @@ -56,6 +59,10 @@ impl TestFunctionExt for Function {
fn is_setup(&self) -> bool {
self.name.is_setup()
}

fn is_fixture(&self) -> bool {
self.name.is_fixture()
}
}

impl TestFunctionExt for String {
Expand All @@ -78,6 +85,10 @@ impl TestFunctionExt for String {
fn is_setup(&self) -> bool {
self.as_str().is_setup()
}

fn is_fixture(&self) -> bool {
self.as_str().is_fixture()
}
}

impl TestFunctionExt for str {
Expand All @@ -100,6 +111,10 @@ impl TestFunctionExt for str {
fn is_setup(&self) -> bool {
self.eq_ignore_ascii_case("setup")
}

fn is_fixture(&self) -> bool {
self.starts_with("fixture")
}
}

/// An extension trait for `std::error::Error` for ABI encoding.
Expand Down
5 changes: 0 additions & 5 deletions crates/config/src/fuzz.rs
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,6 @@ pub struct FuzzDictionaryConfig {
/// Once the fuzzer exceeds this limit, it will start evicting random entries
#[serde(deserialize_with = "crate::deserialize_usize_or_max")]
pub max_fuzz_dictionary_values: usize,
/// How many random addresses to use and to recycle when fuzzing calldata.
/// If not specified then `max_fuzz_dictionary_addresses` value applies.
#[serde(deserialize_with = "crate::deserialize_usize_or_max")]
pub max_calldata_fuzz_dictionary_addresses: usize,
}

impl Default for FuzzDictionaryConfig {
Expand All @@ -127,7 +123,6 @@ impl Default for FuzzDictionaryConfig {
max_fuzz_dictionary_addresses: (300 * 1024 * 1024) / 20,
// limit this to 200MB
max_fuzz_dictionary_values: (200 * 1024 * 1024) / 32,
max_calldata_fuzz_dictionary_addresses: 0,
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions crates/config/src/inline/conf_parser.rs
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,7 @@ mod tests {
function: Default::default(),
line: Default::default(),
docs: r"
forge-config: ciii.invariant.depth = 1
forge-config: ciii.invariant.depth = 1
forge-config: default.invariant.depth = 1
"
.into(),
Expand All @@ -167,7 +167,7 @@ mod tests {
function: Default::default(),
line: Default::default(),
docs: r"
forge-config: ci.invariant.depth = 1
forge-config: ci.invariant.depth = 1
forge-config: default.invariant.depth = 1
"
.into(),
Expand Down
2 changes: 1 addition & 1 deletion crates/config/src/inline/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ impl<T> InlineConfig<T> {
}

/// Inserts an inline configuration, for a test function.
/// Configuration is identified by the pair "contract", "function".
/// Configuration is identified by the pair "contract", "function".
pub fn insert<C, F>(&mut self, contract_id: C, fn_name: F, config: T)
where
C: Into<String>,
Expand Down
7 changes: 5 additions & 2 deletions crates/evm/evm/src/executors/fuzz/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ use foundry_evm_core::{
use foundry_evm_coverage::HitMaps;
use foundry_evm_fuzz::{
strategies::{build_initial_state, fuzz_calldata, fuzz_calldata_from_state, EvmFuzzState},
BaseCounterExample, CounterExample, FuzzCase, FuzzError, FuzzTestResult,
BaseCounterExample, CounterExample, FuzzCase, FuzzError, FuzzFixtures, FuzzTestResult,
};
use foundry_evm_traces::CallTraceArena;
use proptest::test_runner::{TestCaseError, TestError, TestRunner};
Expand Down Expand Up @@ -55,6 +55,7 @@ impl FuzzedExecutor {
pub fn fuzz(
&self,
func: &Function,
fuzz_fixtures: &FuzzFixtures,
address: Address,
should_fail: bool,
rd: &RevertDecoder,
Expand All @@ -80,10 +81,12 @@ impl FuzzedExecutor {
let state = self.build_fuzz_state();

let dictionary_weight = self.config.dictionary.dictionary_weight.min(100);

let strat = proptest::prop_oneof![
100 - dictionary_weight => fuzz_calldata(func.clone()),
100 - dictionary_weight => fuzz_calldata(func.clone(), fuzz_fixtures),
dictionary_weight => fuzz_calldata_from_state(func.clone(), &state),
];

debug!(func=?func.name, should_fail, "fuzzing");
let run_result = self.runner.clone().run(&strat, |calldata| {
let fuzz_res = self.single_fuzz(address, should_fail, calldata)?;
Expand Down
31 changes: 13 additions & 18 deletions crates/evm/evm/src/executors/invariant/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ use foundry_evm_fuzz::{
},
strategies::{
build_initial_state, collect_created_contracts, invariant_strat, override_call_strat,
CalldataFuzzDictionary, EvmFuzzState,
EvmFuzzState,
},
FuzzCase, FuzzedCases,
FuzzCase, FuzzFixtures, FuzzedCases,
};
use foundry_evm_traces::CallTraceArena;
use parking_lot::RwLock;
use proptest::{
strategy::{BoxedStrategy, Strategy, ValueTree},
strategy::{BoxedStrategy, Strategy},
test_runner::{TestCaseError, TestRunner},
};
use revm::{primitives::HashMap, DatabaseCommit};
Expand Down Expand Up @@ -88,12 +88,8 @@ sol! {
}

/// Alias for (Dictionary for fuzzing, initial contracts to fuzz and an InvariantStrategy).
type InvariantPreparation = (
EvmFuzzState,
FuzzRunIdentifiedContracts,
BoxedStrategy<BasicTxDetails>,
CalldataFuzzDictionary,
);
type InvariantPreparation =
(EvmFuzzState, FuzzRunIdentifiedContracts, BoxedStrategy<BasicTxDetails>);

/// Enriched results of an invariant run check.
///
Expand Down Expand Up @@ -152,14 +148,15 @@ impl<'a> InvariantExecutor<'a> {
pub fn invariant_fuzz(
&mut self,
invariant_contract: InvariantContract<'_>,
fuzz_fixtures: &FuzzFixtures,
) -> Result<InvariantFuzzTestResult> {
// Throw an error to abort test run if the invariant function accepts input params
if !invariant_contract.invariant_function.inputs.is_empty() {
return Err(eyre!("Invariant test function should have no inputs"))
}

let (fuzz_state, targeted_contracts, strat, calldata_fuzz_dictionary) =
self.prepare_fuzzing(&invariant_contract)?;
let (fuzz_state, targeted_contracts, strat) =
self.prepare_fuzzing(&invariant_contract, fuzz_fixtures)?;

// Stores the consumed gas and calldata of every successful fuzz call.
let fuzz_cases: RefCell<Vec<FuzzedCases>> = RefCell::new(Default::default());
Expand Down Expand Up @@ -329,7 +326,7 @@ impl<'a> InvariantExecutor<'a> {
Ok(())
});

trace!(target: "forge::test::invariant::calldata_address_fuzz_dictionary", "{:?}", calldata_fuzz_dictionary.inner.addresses);
trace!(target: "forge::test::invariant::fuzz_fixtures", "{:?}", fuzz_fixtures);
trace!(target: "forge::test::invariant::dictionary", "{:?}", fuzz_state.dictionary_read().values().iter().map(hex::encode).collect::<Vec<_>>());

let (reverts, error) = failures.into_inner().into_inner();
Expand All @@ -350,6 +347,7 @@ impl<'a> InvariantExecutor<'a> {
fn prepare_fuzzing(
&mut self,
invariant_contract: &InvariantContract<'_>,
fuzz_fixtures: &FuzzFixtures,
) -> eyre::Result<InvariantPreparation> {
// Finds out the chosen deployed contracts and/or senders.
self.select_contract_artifacts(invariant_contract.address)?;
Expand All @@ -360,16 +358,13 @@ impl<'a> InvariantExecutor<'a> {
let fuzz_state: EvmFuzzState =
build_initial_state(self.executor.backend.mem_db(), self.config.dictionary);

let calldata_fuzz_config =
CalldataFuzzDictionary::new(&self.config.dictionary, &fuzz_state);

// Creates the invariant strategy.
let strat = invariant_strat(
fuzz_state.clone(),
targeted_senders,
targeted_contracts.clone(),
self.config.dictionary.dictionary_weight,
calldata_fuzz_config.clone(),
fuzz_fixtures.clone(),
)
.no_shrink()
.boxed();
Expand All @@ -387,7 +382,7 @@ impl<'a> InvariantExecutor<'a> {
fuzz_state.clone(),
targeted_contracts.clone(),
target_contract_ref.clone(),
calldata_fuzz_config.clone(),
fuzz_fixtures.clone(),
),
target_contract_ref,
));
Expand All @@ -396,7 +391,7 @@ impl<'a> InvariantExecutor<'a> {
self.executor.inspector.fuzzer =
Some(Fuzzer { call_generator, fuzz_state: fuzz_state.clone(), collect: true });

Ok((fuzz_state, targeted_contracts, strat, calldata_fuzz_config))
Ok((fuzz_state, targeted_contracts, strat))
}

/// Fills the `InvariantExecutor` with the artifact identifier filters (in `path:name` string
Expand Down
41 changes: 40 additions & 1 deletion crates/evm/fuzz/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use foundry_evm_coverage::HitMaps;
use foundry_evm_traces::CallTraceArena;
use itertools::Itertools;
use serde::{Deserialize, Serialize};
use std::{collections::HashMap, fmt};
use std::{collections::HashMap, fmt, sync::Arc};

pub use proptest::test_runner::{Config as FuzzConfig, Reason};

Expand Down Expand Up @@ -272,3 +272,42 @@ impl FuzzedCases {
self.lowest().map(|c| c.gas).unwrap_or_default()
}
}

/// Fixtures to be used for fuzz tests.
/// The key represents name of the fuzzed parameter, value holds possible fuzzed values.
/// For example, for a fixture function declared as
/// `function fixture_sender() external returns (address[] memory senders)`
/// the fuzz fixtures will contain `sender` key with `senders` array as value
#[derive(Clone, Default, Debug)]
pub struct FuzzFixtures {
inner: Arc<HashMap<String, DynSolValue>>,
}

impl FuzzFixtures {
pub fn new(fixtures: HashMap<String, DynSolValue>) -> FuzzFixtures {
Self { inner: Arc::new(fixtures) }
}

/// Returns configured fixtures for `param_name` fuzzed parameter.
pub fn param_fixtures(&self, param_name: &str) -> Option<&[DynSolValue]> {
if let Some(param_fixtures) = self.inner.get(&normalize_fixture(param_name)) {
match param_fixtures {
DynSolValue::FixedArray(_) => param_fixtures.as_fixed_array(),
_ => param_fixtures.as_array(),
}
} else {
None
}
}
}

/// Extracts fixture name from a function name.
/// For example: fixtures defined in `fixture_Owner` function will be applied for `owner` parameter.
pub fn fixture_name(function_name: String) -> String {
normalize_fixture(function_name.strip_prefix("fixture").unwrap())
}

/// Normalize fixture parameter name, for example `_Owner` to `owner`.
fn normalize_fixture(param_name: &str) -> String {
param_name.trim_matches(&['_']).to_ascii_lowercase()
}
Loading
Loading