crash in quick mode without "--noplot" when there is no time variation

[KonradHoeffner]

When I benchmark in normal mode, all is fine but with -- --quick I get assertion failed: slice.len() > 1 && slice.iter().all(|x| !x.is_nan()):

cargo bench -- --save-baseline main
...
Benchmarking ??? (all)/0.2 all str triples: Warming up for 3.0000 s
Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 83.1s.
??? (all)/0.2 all str triples
                        time:   [9.8296 s 9.9077 s 9.9875 s]
                        change: [+3531649676% +3559731589% +3589878809%] (p = 0.01 < 0.05)
                        Performance has regressed.

cargo bench -- --quick --save-baseline main
??? (all)/0.2 all str triples
                        time:   [1.0000 ms 1.0000 ms 1.0000 ms]
                        change: [-99.990% -99.990% -99.990%] (p = 0.01 < 0.05)
                        Performance has improved.
thread 'main' panicked at 'assertion failed: slice.len() > 1 && slice.iter().all(|x| !x.is_nan())', /home/konrad/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/criterion-0.4.0/src/stats/univariate/sample.rs:31:9
stack backtrace:
   0: rust_begin_unwind
             at /rustc/246eae2fab54a5139365c4e1a08c5724fb385fbf/library/std/src/panicking.rs:579:5
   1: core::panicking::panic_fmt
             at /rustc/246eae2fab54a5139365c4e1a08c5724fb385fbf/library/core/src/panicking.rs:64:14
   2: core::panicking::panic
             at /rustc/246eae2fab54a5139365c4e1a08c5724fb385fbf/library/core/src/panicking.rs:114:5
   3: criterion::plot::gnuplot_backend::pdf::pdf_small
   4: <criterion::plot::gnuplot_backend::Gnuplot as criterion::plot::Plotter>::pdf
   5: <criterion::html::Html as criterion::report::Report>::measurement_complete
   6: criterion::analysis::common
   7: criterion::benchmark_group::BenchmarkGroup<M>::bench_function
   8: bench::main
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Benchmark code:

let mut group = c.benchmark_group("??? (all)");
group.sample_size(10);  
group.bench_function("0.2 all str triples", |b| b.iter(|| graph.hdt.triples().count()));                       
group.finish();

[KonradHoeffner]

P.S.: The error goes away when I disable plotting with cargo bench -- --quick --noplot but the time is still reported as 1ms, which is incorrect.

[KonradHoeffner]

P.P.S.: This error only occurs when the time goes above around 5 seconds, then it gets reported as 1 ms.
Could this result from arithmetic overflow?
With a smaller dataset, group.bench_function("foo", |b| b.iter(|| for i in 0..32000 {graph.triples().count();})); results in time: [4.9853 s 4.9993 s 5.0028 s] but group.bench_function("foo", |b| b.iter(|| for i in 0..32500 {graph.triples().count();})); gets reported as time: [1.0000 ms 1.0000 ms 1.0000 ms].

[KonradHoeffner]

Minimal Working Example

use criterion::{criterion_group, criterion_main, Criterion};
use std::{thread, time};

fn quick_bug(c: &mut Criterion) {
    let one_second = time::Duration::from_secs(1);  
    let six_seconds = time::Duration::from_secs(6);
    let mut group = c.benchmark_group("> 5 second quick bug test");
    group.sample_size(10);
    group.bench_function("1 seconds", |b| b.iter(|| thread::sleep(one_second)));  
    group.bench_function("6 seconds", |b| b.iter(|| thread::sleep(six_seconds)));
    group.finish();
}

criterion_group!(benches, quick_bug);
criterion_main!(benches);

quickbug$ cargo bench -- --quick
   Compiling quickbug v0.1.0 (/home/konrad/tmp/quickbug)
    Finished bench [optimized] target(s) in 0.71s
     Running benches/bench.rs (target/release/deps/bench-1ea7ae8fa314d649)
> 5 second quick bug test/1 seconds
                        time:   [1.0001 s 1.0001 s 1.0001 s]
> 5 second quick bug test/6 seconds
                        time:   [1.0000 ms 1.0000 ms 1.0000 ms]
thread 'main' panicked at 'assertion failed: slice.len() > 1 && slice.iter().all(|x| !x.is_nan())', /home/konrad/tmp/criterion.rs/src/stats/univariate/sample.rs:31:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[KonradHoeffner]

Minimal Working Example Repository

https://github.com/KonradHoeffner/criterion-quick-bug

[KonradHoeffner]

The error still occurs in version 0.5.0.
While the pull request #659 is now merged in, so you can get the correct times with "--noplot", it still crashes in quick mode when there is no variation in the time when "--noplot" is not specified.
https://github.com/KonradHoeffner/criterion-quick-bug is now updated to 0.5.0 so you can verify this.