Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce weighting of numeric edge cases #455

Closed
wants to merge 1 commit into from

Conversation

ashawley
Copy link
Contributor

@ashawley ashawley commented Feb 13, 2019

Bill Venners recently gave a talk (video is online) about property testing with ScalaTest at an SF Scala meetup, and mentioned a defect he found with Arbitrary[Int] in ScalaCheck. He showed in a repl that ScalaCheck gives equal weight to the edge cases (zero, one, negative one, max, negative max, etc.) as the rest of the numeric range. This weighting causes edge cases to be 50% of the values in a property test with Gen.chooseNum. This risks running tests repeatedly with the same inputs.

+ Gen.chooseNum edge cases: OK, passed 100 tests.
> Collected test data: 
51% non-special
49% special

This pull requests reduces the proportion of edge cases to something more reasonable, just 10% of the time.

+ Gen.chooseNum edge cases: OK, passed 100 tests.
> Collected test data: 
87% non-special
13% special

@ashawley ashawley force-pushed the decrease-edge-cases branch from 4811eaf to 6a89450 Compare February 25, 2019 16:11
@ashawley ashawley mentioned this pull request Feb 25, 2019
18 tasks
@ashawley ashawley force-pushed the decrease-edge-cases branch 2 times, most recently from 429dd5f to 8b29f8c Compare February 27, 2019 20:35
Edge cases are currently 50% of generated values for Gen.chooseNum,
and by consequence for arbitrary Byte, Short, Int and Long.  For
property tests, this produces a lot of redundant testing for the
same inputs.
@ashawley ashawley force-pushed the decrease-edge-cases branch from 8b29f8c to 72cd8fa Compare March 4, 2019 17:05
@ashawley ashawley changed the title Fix defect with Arbitrary[Int] edge cases Reduce weighting of numeric edge cases Mar 24, 2019
@ashawley
Copy link
Contributor Author

One argument against this change concerns combined probabilities of independent events.

For example, if you generate edge cases 10% of the time for a numeric value instead of 50%, then when you need two numeric values you generate edge cases for both values 1% of the time (0.1 * 0.1) rather than 25% of the time (0.5 * 0.5).

The rest of the time, 99% of cases, you'd be generating two values where neither are edge cases 81% of the time (0.9 * 0.9) or the two values are an edge case and non-edge case 18% of the time (2 * .9 * .01).

In isolation, repeats of edge cases look bad. However, "fixing" it has other consequences when generating tuples. Ideally, ScalaCheck could compensate for conditional probability when weighting edge cases for tuples.

@ashawley
Copy link
Contributor Author

In light of my previous comment on Mar-24 and now #485 about Option, it seems that changing the weighting of things has unintended consequences. This could be revisited and taken up at another time.

@ashawley ashawley closed this Jul 17, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant