Clarify behaviour and interplay of --run-types, --tests, --doc, --all-targets, etc.

[alexpovel]

Came here in trying to find out how to run overall coverage, aka doc, unit and perhaps more tests combined.

Initially I found this comment in #13 which says:

tarpaulin now has a new --run-type flag where you can get coverage for just your tests (default), just doc-tests or both!

I'm not sure how to get this working. Perhaps things changed in the four years since. #850 is tightly related but the below differs, and I didn't want to comment in an old, closed issue.

I am working with:

$ cargo tarpaulin --version
cargo-tarpaulin version: 0.25.2

and have a sample project:

$ cargo new --lib tarpaulin-test
     Created library `tarpaulin-test` package
$ tree tarpaulin-test
tarpaulin-test
├── Cargo.toml
└── src
    └── lib.rs

with the lib.rs as:

pub fn one() {
    println!("one");
}

/// # Examples
/// ```
/// use tarpaulin_test::two;
/// two();
/// ```
pub fn two() {
    println!("two");
}

mod tests {
    use super::*;

    #[test]
    fn test_one() {
        one();
    }
}

and Cargo.toml:

[package]
name = "tarpaulin-test"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html

[dependencies]

Now, cargo tarpaulin --doc reports 50%, cargo tarpaulin --tests reports 50% as well, and lastly cargo tarpaulin --doc --tests gives 100% coverage. Great, works! But how does --run-type AllTargets work? Its behaviour isn't clear to me:

• cargo tarpaulin --run-types doctests tests only doctests, which is fine. Reports 50% coverage.
• cargo tarpaulin --run-types tests tests only unit tests, also fine. Gives 50% coverage.
• cargo tarpaulin --run-types AllTargets, contrary to the above cargo tarpaulin --doc --tests, exercises only the unit, not the doc test, giving 50% as well, which is unexpected.

Run the above examples with --out html to see the coverage "flip around" between the one unit and the one doc test.

Another thing of note is how the help output is written, where it says:

$ cargo tarpaulin --help | grep -- '--doc'
        --doc                      Test only this library's documentation

but also, for example:

$ cargo tarpaulin --help | grep -- '--examples'
        --examples                 Test all examples

As far as I understand one could run cargo tarpaulin --tests --doc --examples to run all three and combine results. However, only for the --doc option does it say only, which I guess is misleading?

Lastly, either a hint that these options combine (which ones exactly then? --tests, --doc, --examples, ... --benches I guess? Did I miss anything?) would be helpful, and/or a flag for running everything combined (is this was --run-types AllTargets is designed for)?

I can offer looking into clearing this up in a PR, but the --run-types arg I'm not clear on yet. It seems related to other, 'single' flags:

tarpaulin/src/cargo.rs

Lines 535 to 544 in 8c876b1

	if let Some(ty) = ty {
	match ty {
	RunType::Tests => test_cmd.arg("--tests"),
	RunType::Doctests => test_cmd.arg("--doc"),
	RunType::Benchmarks => test_cmd.arg("--benches"),
	RunType::Examples => test_cmd.arg("--examples"),
	RunType::AllTargets => test_cmd.arg("--all-targets"),
	RunType::Lib => test_cmd.arg("--lib"),
	RunType::Bins => test_cmd.arg("--bins"),
	};

In fact, passing just --all-targets errors out, which is quite unexpected behaviour on the CLI (passing a single flag shouldn't error out, as it can't contradict anything; maybe ignore if tarpaulin already passes it to cargo test?):

$ cargo tarpaulin --all-targets
Jun 12 18:55:24.481  INFO cargo_tarpaulin::config: Creating config
Jun 12 18:55:24.487  INFO cargo_tarpaulin: Running Tarpaulin
Jun 12 18:55:24.487  INFO cargo_tarpaulin: Building project
Jun 12 18:55:24.487  INFO cargo_tarpaulin::cargo: Cleaning project
error: the argument '--all-targets' cannot be used multiple times

Usage: cargo test [OPTIONS] [TESTNAME] [-- [args]...]

For more information, try '--help'.
Jun 12 18:55:24.496 ERROR cargo_tarpaulin: Cargo failed to run! Error: cargo run failed
Error: "Cargo failed to run! Error: cargo run failed"

The help output doesn't warn of this potential behaviour:

$ cargo tarpaulin --help | grep -C3 -- '--all-targets'
FLAGS:
        --all                      Alias for --workspace (deprecated)
        --all-features             Build all available features
        --all-targets              Test all targets
        --avoid-cfg-tarpaulin      Remove --cfg=tarpaulin from the RUSTFLAG
        --benches                  Test all benches
        --bins                     Test all binaries

Thanks for putting together this crate, it works really well! Just at this point I reckon the docs can be tad clearer.

[xd009642]

So tarpaulin actually matches cargo tests behaviour in regards to this - which meant I had to look it up because my help text for it is also taken verbatim from cargo test. From https://doc.rust-lang.org/cargo/commands/cargo-test.html

--all-targets
    Test all targets. This is equivalent to specifying --lib --bins --tests --benches --examples.

Allowing --doc --examples to be args is where tarpaulin differs - probably because I felt at the time it made sense for these to be additive.

The error with all-targets is likely due to over-enthusiastically passing flags into cargo test when building the test binaries.

Hopefully this clears up some of the initial questions or provides some clarity. Tomorrow I'll try to find some time to go over in more detail and actually address the issues in the docs and make --all-targets work. Though I probably won't add doc tests to it because doc test coverage does rely on nightly features

[alexpovel]

Thanks for the fast reply! The help for cargo test actually doesn't contain that bit on my system, so no wonder I wasn't able to find that info:

$ cargo test --help | grep -C2 -- '--all-targets'
      --benches                 Test all benches
      --offline                 Run without accessing the network
      --all-targets             Test all targets
      --config <KEY=VALUE>      Override a configuration value
      --doc                     Test only this library's documentation

so that would've helped clear up the confusion. Perhaps a version mismatch (I have v1.69.0).

It's helpful to see that tarpaulin mirrors that, thanks. I agree about being pragmatic in the --doc case (have it be additive).

[xd009642]

Yeah so the help for cargo test doesn't include it, it's only in the cargo book hosted online. Maybe worth raising an issue on rust-lang/cargo if you feel the CLI help text could stand to be improved. I'm adding (excluding doctests) to my own help text to clear up the ambiguity.

I've also tracked down the issue with --all-targets so a PR to fix that should be landing today or tomorrow and a new release before the end of the week!