assertion failed: st.current_test_count() == st.total when running benchmarks

[philipc]

I tried running benchmarks with current nightly rust:

cargo +nightly bench bench_evaluating_debug_info_expressions
    Finished bench [optimized + debuginfo] target(s) in 0.01s
     Running benches/bench.rs (target/release/deps/bench-ca766c0cc6658384)

running 0 tests
test bench_evaluating_debug_info_expressions                   ... bench:     393,281 ns/iter (+/- 10,673)
thread 'main' panicked at 'assertion failed: st.current_test_count() == st.total', library/test/src/console.rs:296:5
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: bench failed, to rerun pass `--bench bench`

Running with the previous nightly works:

cargo +nightly-2022-10-30 bench bench_evaluating_debug_info_expressions
    Finished bench [optimized + debuginfo] target(s) in 0.01s
     Running benches/bench.rs (target/release/deps/bench-ca766c0cc6658384)

running 1 test
test bench_evaluating_debug_info_expressions                   ... bench:     391,268 ns/iter (+/- 6,342)

test result: ok. 0 passed; 0 failed; 0 ignored; 1 measured; 22 filtered out; finished in 0.71s

Also, note the difference in the line running 0 tests.

This happens for the benchmarks in both the gimli and addr2line crates.

#103689 looks to be related (@saethlin)

Meta

rustc +nightly --version --verbose
rustc 1.67.0-nightly (77e57db38 2022-10-30)
binary: rustc
commit-hash: 77e57db384aca99444c3b5f6a9c86bc58a804d89
commit-date: 2022-10-30
host: x86_64-unknown-linux-gnu
release: 1.67.0-nightly
LLVM version: 15.0.4

Backtrace

stack backtrace:
   0: rust_begin_unwind
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/std/src/panicking.rs:575:5
   1: core::panicking::panic_fmt
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/core/src/panicking.rs:65:14
   2: core::panicking::panic
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/core/src/panicking.rs:115:5
   3: test::console::run_tests_console
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/test/src/console.rs:296:5
   4: test::test_main
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/test/src/lib.rs:140:15
   5: test::test_main_static
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/test/src/lib.rs:159:5
   6: call_once<fn(), ()>
             at /rustc/77e57db384aca99444c3b5f6a9c86bc58a804d89/library/core/src/ops/function.rs:251:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[saethlin]

Ouch. I guess libtest needs better tests. I might be able to get to this in 12 hours or so, if anyone else sees this please do take care of it.

[thomcc]

Oof, missed that filtered_tests started as the full list of tests+benches before getting shadowed as only the tests. Not a fan of the way TestEvent is used to manage state (haven't ever been a fan of it, TBH), but I'll have to think a bit about how to fix it without undoing the perf wins we just got here.

[bwsw]

Confirm the bug as well:
https://github.com/insight-platform/Similari
cargo bench started to fail after changing the toolset to the current nighly.

[thomcc]

Yeah, should be fixed by #103795. Hopefully will be in the next nightly, if we're lucky.