Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(fuzz): ability to declare fuzz test fixtures #7428

Merged
merged 31 commits into from
Apr 22, 2024
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
ef15bab
fix(fuzz): deduplicate fuzz inputs
grandizzy Mar 14, 2024
3c48666
Fix tests, collect fixtures in test setup, arc fixtures
grandizzy Mar 19, 2024
e1f8a95
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 19, 2024
254017c
Cleanup
grandizzy Mar 19, 2024
8586bf4
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 20, 2024
14c869a
Use fixture_ prefix
grandizzy Mar 20, 2024
d0442de
Update tests to reflect that random values are used if no fixtures
grandizzy Mar 20, 2024
7e4b14c
Review changes
grandizzy Mar 21, 2024
972d993
Group fuzz_calldata and fuzz_calldata_from_state in calldata mod
grandizzy Mar 21, 2024
708e4db
Review changes: remove unnecessary clones, nicer code to collect fixt…
grandizzy Mar 21, 2024
f93254a
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 22, 2024
9276893
Add support for bytes and string fixtures, fixture strategy macro. So…
grandizzy Mar 24, 2024
a5e8da0
Remove unnecessary clone
grandizzy Mar 26, 2024
14273fa
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 26, 2024
09ee66e
Use inline config
grandizzy Mar 26, 2024
342eaf7
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Mar 26, 2024
4785f93
More robust invariant assume test
grandizzy Mar 26, 2024
0b7a230
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 5, 2024
f9adb66
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 10, 2024
fb86084
Fixtures as storage arrays, remove inline config
grandizzy Apr 10, 2024
6874ead
Simplify code
grandizzy Apr 10, 2024
ca4d44b
Support fixed size arrays fixtures
grandizzy Apr 10, 2024
acdf92d
Update comment
grandizzy Apr 11, 2024
12115fa
Use DynSolValue::type_strategy for address and fixed bytes fuzzed params
grandizzy Apr 11, 2024
e76fe8e
Add prefix to mark a storage array or a function as fixture
grandizzy Apr 11, 2024
3ac9e33
Fix test
grandizzy Apr 11, 2024
7e95208
Simplify code / fixture strategy macro, panic if configured fixture n…
grandizzy Apr 11, 2024
4cf19cb
Consistent panic with fixture strategy if uint / int fixture of diffe…
grandizzy Apr 13, 2024
53c64d3
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 17, 2024
997c5d4
Review changes: don't panic when invalid fixture, use prop_filter_map…
grandizzy Apr 18, 2024
a127780
Merge remote-tracking branch 'origin' into issue-3521-fixtures
grandizzy Apr 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions crates/common/src/traits.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,9 @@ pub trait TestFunctionExt {

/// Returns whether this function is a `setUp` function.
fn is_setup(&self) -> bool;

/// Returns whether this function is a `fixture` function.
fn is_fixture(&self) -> bool;
}

impl TestFunctionExt for Function {
Expand All @@ -56,6 +59,10 @@ impl TestFunctionExt for Function {
fn is_setup(&self) -> bool {
self.name.is_setup()
}

fn is_fixture(&self) -> bool {
self.name.is_fixture()
}
}

impl TestFunctionExt for String {
Expand All @@ -78,6 +85,10 @@ impl TestFunctionExt for String {
fn is_setup(&self) -> bool {
self.as_str().is_setup()
}

fn is_fixture(&self) -> bool {
self.as_str().is_fixture()
}
}

impl TestFunctionExt for str {
Expand All @@ -100,6 +111,10 @@ impl TestFunctionExt for str {
fn is_setup(&self) -> bool {
self.eq_ignore_ascii_case("setup")
}

fn is_fixture(&self) -> bool {
self.starts_with("fixture_")
}
}

/// An extension trait for `std::error::Error` for ABI encoding.
Expand Down
5 changes: 0 additions & 5 deletions crates/config/src/fuzz.rs
Original file line number Diff line number Diff line change
Expand Up @@ -111,10 +111,6 @@ pub struct FuzzDictionaryConfig {
/// Once the fuzzer exceeds this limit, it will start evicting random entries
#[serde(deserialize_with = "crate::deserialize_usize_or_max")]
pub max_fuzz_dictionary_values: usize,
/// How many random addresses to use and to recycle when fuzzing calldata.
/// If not specified then `max_fuzz_dictionary_addresses` value applies.
#[serde(deserialize_with = "crate::deserialize_usize_or_max")]
pub max_calldata_fuzz_dictionary_addresses: usize,
}

impl Default for FuzzDictionaryConfig {
Expand All @@ -127,7 +123,6 @@ impl Default for FuzzDictionaryConfig {
max_fuzz_dictionary_addresses: (300 * 1024 * 1024) / 20,
// limit this to 200MB
max_fuzz_dictionary_values: (200 * 1024 * 1024) / 32,
max_calldata_fuzz_dictionary_addresses: 0,
}
}
}
Expand Down
7 changes: 5 additions & 2 deletions crates/evm/evm/src/executors/fuzz/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use foundry_evm_fuzz::{
build_initial_state, collect_state_from_call, fuzz_calldata, fuzz_calldata_from_state,
EvmFuzzState,
},
BaseCounterExample, CounterExample, FuzzCase, FuzzError, FuzzTestResult,
BaseCounterExample, CounterExample, FuzzCase, FuzzError, FuzzFixtures, FuzzTestResult,
};
use foundry_evm_traces::CallTraceArena;
use proptest::test_runner::{TestCaseError, TestError, TestRunner};
Expand Down Expand Up @@ -58,6 +58,7 @@ impl FuzzedExecutor {
pub fn fuzz(
&self,
func: &Function,
fuzz_fixtures: &FuzzFixtures,
address: Address,
should_fail: bool,
rd: &RevertDecoder,
Expand All @@ -83,10 +84,12 @@ impl FuzzedExecutor {
let state = self.build_fuzz_state();

let dictionary_weight = self.config.dictionary.dictionary_weight.min(100);

let strat = proptest::prop_oneof![
100 - dictionary_weight => fuzz_calldata(func.clone()),
100 - dictionary_weight => fuzz_calldata(func.clone(), fuzz_fixtures),
dictionary_weight => fuzz_calldata_from_state(func.clone(), &state),
];

debug!(func=?func.name, should_fail, "fuzzing");
let run_result = self.runner.clone().run(&strat, |calldata| {
let fuzz_res = self.single_fuzz(&state, address, should_fail, calldata)?;
Expand Down
31 changes: 13 additions & 18 deletions crates/evm/evm/src/executors/invariant/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,14 @@ use foundry_evm_fuzz::{
},
strategies::{
build_initial_state, collect_created_contracts, collect_state_from_call, invariant_strat,
override_call_strat, CalldataFuzzDictionary, EvmFuzzState,
override_call_strat, EvmFuzzState,
},
FuzzCase, FuzzedCases,
FuzzCase, FuzzFixtures, FuzzedCases,
};
use foundry_evm_traces::CallTraceArena;
use parking_lot::{Mutex, RwLock};
use proptest::{
strategy::{BoxedStrategy, Strategy, ValueTree},
strategy::{BoxedStrategy, Strategy},
test_runner::{TestCaseError, TestRunner},
};
use revm::{primitives::HashMap, DatabaseCommit};
Expand Down Expand Up @@ -88,12 +88,8 @@ sol! {
}

/// Alias for (Dictionary for fuzzing, initial contracts to fuzz and an InvariantStrategy).
type InvariantPreparation = (
EvmFuzzState,
FuzzRunIdentifiedContracts,
BoxedStrategy<BasicTxDetails>,
CalldataFuzzDictionary,
);
type InvariantPreparation =
(EvmFuzzState, FuzzRunIdentifiedContracts, BoxedStrategy<BasicTxDetails>);

/// Enriched results of an invariant run check.
///
Expand Down Expand Up @@ -152,14 +148,15 @@ impl<'a> InvariantExecutor<'a> {
pub fn invariant_fuzz(
&mut self,
invariant_contract: InvariantContract<'_>,
fuzz_fixtures: &FuzzFixtures,
) -> Result<InvariantFuzzTestResult> {
// Throw an error to abort test run if the invariant function accepts input params
if !invariant_contract.invariant_function.inputs.is_empty() {
return Err(eyre!("Invariant test function should have no inputs"))
}

let (fuzz_state, targeted_contracts, strat, calldata_fuzz_dictionary) =
self.prepare_fuzzing(&invariant_contract)?;
let (fuzz_state, targeted_contracts, strat) =
self.prepare_fuzzing(&invariant_contract, fuzz_fixtures)?;

// Stores the consumed gas and calldata of every successful fuzz call.
let fuzz_cases: RefCell<Vec<FuzzedCases>> = RefCell::new(Default::default());
Expand Down Expand Up @@ -328,7 +325,7 @@ impl<'a> InvariantExecutor<'a> {
Ok(())
});

trace!(target: "forge::test::invariant::calldata_address_fuzz_dictionary", "{:?}", calldata_fuzz_dictionary.inner.addresses);
trace!(target: "forge::test::invariant::fuzz_fixtures", "{:?}", fuzz_fixtures);
trace!(target: "forge::test::invariant::dictionary", "{:?}", fuzz_state.read().values().iter().map(hex::encode).collect::<Vec<_>>());

let (reverts, error) = failures.into_inner().into_inner();
Expand All @@ -349,6 +346,7 @@ impl<'a> InvariantExecutor<'a> {
fn prepare_fuzzing(
&mut self,
invariant_contract: &InvariantContract<'_>,
fuzz_fixtures: &FuzzFixtures,
) -> eyre::Result<InvariantPreparation> {
// Finds out the chosen deployed contracts and/or senders.
self.select_contract_artifacts(invariant_contract.address)?;
Expand All @@ -368,16 +366,13 @@ impl<'a> InvariantExecutor<'a> {
let targeted_contracts: FuzzRunIdentifiedContracts =
Arc::new(Mutex::new(targeted_contracts));

let calldata_fuzz_config =
CalldataFuzzDictionary::new(&self.config.dictionary, &fuzz_state);

// Creates the invariant strategy.
let strat = invariant_strat(
fuzz_state.clone(),
targeted_senders,
targeted_contracts.clone(),
self.config.dictionary.dictionary_weight,
calldata_fuzz_config.clone(),
fuzz_fixtures.clone(),
)
.no_shrink()
.boxed();
Expand All @@ -395,7 +390,7 @@ impl<'a> InvariantExecutor<'a> {
fuzz_state.clone(),
targeted_contracts.clone(),
target_contract_ref.clone(),
calldata_fuzz_config.clone(),
fuzz_fixtures.clone(),
),
target_contract_ref,
));
Expand All @@ -404,7 +399,7 @@ impl<'a> InvariantExecutor<'a> {
self.executor.inspector.fuzzer =
Some(Fuzzer { call_generator, fuzz_state: fuzz_state.clone(), collect: true });

Ok((fuzz_state, targeted_contracts, strat, calldata_fuzz_config))
Ok((fuzz_state, targeted_contracts, strat))
}

/// Fills the `InvariantExecutor` with the artifact identifier filters (in `path:name` string
Expand Down
26 changes: 25 additions & 1 deletion crates/evm/fuzz/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use foundry_evm_coverage::HitMaps;
use foundry_evm_traces::CallTraceArena;
use itertools::Itertools;
use serde::{Deserialize, Serialize};
use std::{collections::HashMap, fmt};
use std::{collections::HashMap, fmt, sync::Arc};

pub use proptest::test_runner::{Config as FuzzConfig, Reason};

Expand Down Expand Up @@ -272,3 +272,27 @@ impl FuzzedCases {
self.lowest().map(|c| c.gas).unwrap_or_default()
}
}

/// Fixtures to be used for fuzz tests.
/// The key represents name of the fuzzed parameter, value holds possible fuzzed values.
/// For example, for a fixture function declared as
/// `function fixture_sender() external returns (address[] memory senders)`
/// the fuzz fixtures will contain `sender` key with `senders` array as value
#[derive(Clone, Default, Debug)]
pub struct FuzzFixtures {
inner: Arc<HashMap<String, DynSolValue>>,
}

impl FuzzFixtures {
pub fn new(fixtures: HashMap<String, DynSolValue>) -> FuzzFixtures {
Self { inner: Arc::new(fixtures) }
}

pub fn param_fixtures(&self, param_name: &String) -> Option<&[DynSolValue]> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs oneline doc

if let Some(param_fixtures) = self.inner.get(param_name) {
param_fixtures.as_array()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will now only parse dynamic arrays, let's support fixed-sized arrays as well

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

added with ca4d44b

} else {
None
}
}
}
53 changes: 53 additions & 0 deletions crates/evm/fuzz/src/strategies/address.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
use alloy_dyn_abi::DynSolValue;
use alloy_primitives::Address;
use proptest::{
arbitrary::any,
prelude::{prop, BoxedStrategy},
strategy::Strategy,
};

/// The address strategy combines 2 different strategies:
/// 1. A random addresses strategy if no fixture defined for current parameter.
/// 2. A fixture based strategy if configured values for current parameters.
/// If fixture is not a valid type then an error is raised and test suite will continue to execute
/// with random address.
///
///
/// For example:
/// To define fixture for `owner` fuzzed parameter, return an array of possible values from
/// `function fixture_owner() public returns (address[] memory)`.
/// Use `owner` named parameter in fuzzed test in order to create a custom strategy
/// `function testFuzz_ownerAddress(address owner, uint amount)`.
#[derive(Debug)]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[derive(Debug)]
#[derive(Debug, Default)]
#[non_exhaustive]

pub struct AddressStrategy {}

impl AddressStrategy {
/// Create a new address strategy.
pub fn init(fixtures: Option<&[DynSolValue]>) -> BoxedStrategy<DynSolValue> {
if let Some(fixtures) = fixtures {
let address_fixtures: Vec<DynSolValue> =
fixtures.iter().enumerate().map(|(_, value)| value.to_owned()).collect();
let address_fixtures_len = address_fixtures.len();
any::<prop::sample::Index>()
.prop_map(move |index| {
// Generate value tree from fixture.
// If fixture is not a valid address, raise error and generate random value.
let index = index.index(address_fixtures_len);
if let Some(addr_fixture) = address_fixtures.get(index) {
if let Some(addr_fixture) = addr_fixture.as_address() {
return DynSolValue::Address(addr_fixture);
}
}
error!(
"{:?} is not a valid address fixture, generate random value",
address_fixtures.get(index)
);
DynSolValue::Address(Address::random())
})
.boxed()
} else {
// If no config for addresses dictionary then create unbounded addresses strategy.
any::<Address>().prop_map(DynSolValue::Address).boxed()
}
}
}
Loading
Loading