Testing of distributions

[pitdicker]

We should add some testing for distributions. Unfortunately since distribution samples are random, this is inherently difficult. Original issue dhardy#72.

See for some direction rust-lang/rust#10084, and the random-tests.

[vks]

Also see #290.

[dhardy]

@vks has some plans for histogram-based testing, though possibly only after PDFs are added.

Separately, we should add some tests of value-stability.

[vks]

@vks has some plans for histogram-based testing, though possibly only after PDFs are added.

See tests/uniformity.rs, which could be generalized to arbitrary distributions (with given PDFs).

[dhardy]

@vks could I interest you (or potentially someone else) in adding at utility to rand_distr (perhaps an example) which plots histograms of each distribution from actual samples (not PDFs)? This would at least let us eye-ball that distributions are doing roughly the right thing.

[vks]

I was thinking about using sparklines for that. I'm interested, but I can't make promises that I'll get to it before the end of the year.

[vks]

Resolving this issue was started in #1121, more work is needed to extend it to the other distributions.

[vks]

I think we should either make this issue more actionable or close it.

For which distributions do we want histogram tests? Sometimes, calculating the PDF is non-trivial and requires special functions.

[dhardy]

I suggest the following approach:

1. Pick a representative parameter set (or possibly a small number of parametrisations).
2. Generate a histogram with this parametrisation and manually verify that the result looks reasonable.
3. Potentially lower the number of buckets and samples used by the histogram, and check the result is still similar. (We want tests to be reasonably fast to run.)
4. The test should enforce exact reproduction of this histogram.

This should allow fast and robust testing (within the limits of reproducibility), with the caveat that any value-breaking change must again be visually confirmed.

It should be applicable to all distributions, though may be hard to visualise for multi-dimensional outputs.

It cannot confirm perfect accuracy, especially in degenerate cases.