Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement is_minimal_case #477

Closed
wants to merge 2 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions proptest/src/is_minimal_case.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
use core::cell::Cell;

thread_local! {
static IS_MINIMAL_CASE: Cell<bool> = Cell::new(false);
}

/// When run inside a property test, indicates whether the current case being tested
/// is the minimal test case.
///
/// `proptest` typically runs a large number of test cases for each
/// property test. If it finds a failing test case, it tries to shrink it
/// in the hopes of finding a simpler test case. When debugging a failing
/// property test, we are often only interested in the actual minimal
/// failing case. After the minimal test case has been identified,
/// the test is rerun with the minimal input, and this function
/// returns `true` when called inside the test.
///
/// The results are undefined if property tests are nested, meaning that a property test
/// is run inside another property test.
///
/// # Example
///
/// ```rust
/// use proptest::{proptest, prop_assert, is_minimal_case};
/// # fn export_to_file_for_analysis() {}
///
/// proptest! {
/// #[test]
/// fn test_is_not_five(num in 0 .. 10) {
/// if is_minimal_case() {
/// eprintln!("Minimal test case is {num:?}");
/// export_to_file_for_analysis(num);
/// }
///
/// prop_assert!(num != 5);
/// }
/// }
/// ```
pub fn is_minimal_case() -> bool {
IS_MINIMAL_CASE.with(|cell| cell.get())
}

/// Helper struct that helps to ensure panic safety when entering a minimal case.
///
/// Specifically, if the test case panics, we must ensure that we still
/// correctly reset the thread-local variable.
#[non_exhaustive]
pub(crate) struct MinimalCaseGuard;

impl MinimalCaseGuard {
pub(crate) fn begin_minimal_case() -> Self {
IS_MINIMAL_CASE.with(|cell| cell.replace(true));
Self
}
}

impl Drop for MinimalCaseGuard {
fn drop(&mut self) {
IS_MINIMAL_CASE.with(|cell| cell.replace(false));
}
}
3 changes: 3 additions & 0 deletions proptest/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -54,6 +54,9 @@ mod product_frunk;
#[macro_use]
mod product_tuple;

mod is_minimal_case;
pub use is_minimal_case::is_minimal_case;

#[macro_use]
extern crate bitflags;
#[cfg(feature = "bit-set")]
Expand Down
26 changes: 24 additions & 2 deletions proptest/src/test_runner/runner.rs
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ use std::env;
use std::fs;
#[cfg(feature = "fork")]
use tempfile;

use crate::is_minimal_case::MinimalCaseGuard;
use crate::strategy::*;
use crate::test_runner::config::*;
use crate::test_runner::errors::*;
Expand Down Expand Up @@ -735,13 +735,35 @@ impl TestRunner {
let why = self
.shrink(
&mut case,
test,
&test,
replay_from_fork,
result_cache,
fork_output,
is_from_persisted_seed,
)
.unwrap_or(why);

// Run minimal test again
let _guard = MinimalCaseGuard::begin_minimal_case();
let minimal_result = call_test(
self,
case.current(),
&test,
&mut iter::empty(),
&mut *noop_result_cache(),
// TODO: What should fork_output be?
fork_output,
is_from_persisted_seed,
);

if !matches!(minimal_result, Err(TestCaseError::Fail(_))) {
// TODO: Is it appropriate to use eprintln! here?
// It seems appropriate to atleast notify the user somehow
// that the minimal test case does not consistently fail
eprintln!("unexpected behavior: minimal case did not result in test \
failure on second test run");
}

Err(TestError::Fail(why, case.current()))
}
Err(TestCaseError::Reject(whence)) => {
Expand Down
Loading