Support discoverability of test targets accepted by cargo test --test

[rrnewton]

Problem

Context: I have many test targets and want to narrow down the scope of cargo test while in the development loop. Regular test filtering still spends a bunch of time running targets that say ok. 0 passed.

This interaction is frustrating:

$ cargo test --test foobar
error: no test target named `foobar`

Ok, great, but please tell me what the set of test targets is or how to find it. In this case, I want to run an integration test corresponding to a file tests/foobar.rs, but not how to express that in the right format for --test. Let's check the help:

$ cargo test --help
...
        --test <NAME>...                             Test only the specified test target

That doesn't shine any more light on it. What is the set from which NAME is drawn? What's the "type", if you will?

Next I looked for something like --list-targets but didn't find it.

Finally, I look through the output of cargo test, hoping that it essentially serves the function of listing test targets. But the printouts during a test run look like this:

     Running target/debug/deps/foobar-ed66a800a301dfb1

And that's clearly not the string I'm supposed to provide to --test!

Solution

Here are three possible solutions:

1. Provide a cargo test --list and mention it from the cargo test --test error message.
2. Print out the list of literal target strings as part of the cargo test --test error message. This can be truncated if it's excessively long.
3. Print precise target strings in a clear way while running tests.

Related

It seems #8396 is the only other issue that mentions this error message.
There was also this very old #812 discussing the documentation for cargo test vs cargo test --test.

[epage]

cargo test --test will print out the list of tests but this isn't mentioned by cargo test -h or cargo help test.

So I can see

• Having cargo test --test <name> provide suggestions like clap does
• Document cargo test --test

The downside to cargo test --list is that we specifically want the list for cargo test --test but a --list flag doesn't make the clear compared to any of the other types of tests that can be run.

[sanmai-NL]

@epage That invocation doesn't work in my case (cargo 1.66.0 (d65d197ad 2022-11-15) on Linux inside a workspace).

sanmai@server4:~/workspace$ cargo test --no-run --test
error: "--test" takes one argument.
No tests available.

sanmai@server4:~/workspace$ cargo test --test
error: "--test" takes one argument.
No tests available.

[epage]

@sanmai-NL

That invocation doesn't work in my case (cargo 1.66.0 (d65d197 2022-11-15) on Linux inside a workspace).

Not sure what to say but it works for me in a workspace (virtual or not) or out of it

$ cargo -V
cargo 1.68.0 (115f34552 2023-02-26)
$ cargo test --test
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    ui
$ cargo +1.64.0 test --test
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    ui
$ cargo +1.60.0 test --test  # Note: I had to switch to an older version with an appropriate MSRV
error: "--test" takes one argument.
Available tests:
    builder
    derive
    derive_ui
    examples
    macros
    yaml

[epage]

@sanmai-NL I just noticed your error said No tests available.. Sounds like this is something related to your project. You either don't have any test binaries (only --lib or --doc) or there is some enumeration bug for your project and a minimal reproduction case would be needed

[sanmai-NL]

The workspace isn't mine. But say it only has lib tests, then this proves that command isn't generic for test function discovery. The nextest command is (for lib tests at least).

[epage]

This issue is about discovery of test targets usable with cargo test --test which was the context for my answer. If you are looking at enumerating all test binaries to enumerate all test functions, that is different and my answer will not help with that.

[sanmai-NL]

@epage I wasn't fully aware of the distinction in libtest of these 'integration tests' and other types of tests. Just was aware of doctests, benches and functions with the test attribute. (Quoted because defining automated testing semantics through structure is IMO quite ... restricting.)

@rrnewton Would you mind clarifying the title to refer to integration tests rather than the current more technical description? Helps understandability and findability of the issue.

So, I think the more general functional requirement would be something like: as a developer or test engineer in Rust, want a CLI report in terms of a machine readable list of all discovered tests by their full item paths. Here, test meaning every function that would be executed when cargo test is run on a crate or workspace given the libtest test harness.

If we want to broaden the issue, then suggest to change the title as well.

[epage]

So, I think the more general functional requirement would be something like: as a developer or test engineer in Rust, want a CLI report in terms of a machine readable list of all discovered tests by their full item paths. Here, test meaning every function that would be executed when cargo test is run on a crate or workspace given the libtest test harness.

If we want to broaden the issue, then suggest to change the title as well.

No, this is s a very different issue. The current issue is about making it easier to discover the appropriate parameters to the cargo test --test option.

If you are wanting something else, please open a different issue. However, I suspect what you want is already available because cargo nextest is able to enumerate all tests before individually kicking them off. The main issue is the json output for libtest is not stable, see rust-lang/rust#49359.