Implement support for LLVMs code coverage instrumentation

[gnzlbg]

There are ways to more or less easily obtain code coverage information from rust binaries, some of which are in widespread use (e.g. travis-cargo + gcov + coveralls.io). However, these are either platform specific (gcov/kcov are linux only), or incomplete (coverage for documentation tests is not collected).

It would be better if rustc would be able to instrument rust binaries and tests using LLVM Coverage Instrumentation. This would allow code coverage information to work portably across the supported platforms, as well as produce reliable coverage information.

Ideally, such support would be integrated in an easy to use cargo coverage subcommand and the Rust Language Server, such that IDEs can use this information to guide the user while writing tests.

[alexcrichton]

I know I'd love to see this work! Do you know what it would entail in terms of what the hoops a prototype would have to jump through? Also, does this work reliably on platforms like Windows? Or is it still best on Linux and "works mostly elsewhere"?

[gnzlbg]

A prototype would need to generate the LLVM IR for the instrumentation, for that it needs to generate a mapping between source ranges and performance counters (I would start with just functions). Then it needs to generate and embed the IR in the object files. It would be a good idea to look how clang then outputs this into a file to use the same format (which is documented), and test that we can read it with llvm-cov in linux and macos (don't know about windows support but since clang is gaining full windows support if its not there it will be there soon).

I think that would be enough for a prototype, from there we can move on to generating instrumentation for conditionals (match/loop/jumps/ifs...) and blocks (how many iterations was a loop body executed). We could go down to expressions, but then it would be wise to offer users a way to control how much instrumentation is generated: functions, branches, and loops (which are branches) is typically enough. We should then support skipped regions (for conditional compilation), expansions (for macros, maybe plugins), and dealing with unreachable code (there is a performance counter that is always zero for that).

The meat of the work is in the source=> performance counter mapping, generating the IR, and generating the conforming output. llvm-cov works at least on mac and linux, but IIRC clang has options to generate coverage information even for particular gcov versions. The source code for all this is available so taking a look wouldn't hurt.

Also, does this work reliably on platforms like Windows? Or is it still best on Linux and "works mostly elsewhere"?

It works on Mac and Linux, don't know about Windows. Even if it doesn't work everywhere, this is probably the way to make it work in as much platforms as possible anyways. Clang support on windows is pretty good already and it is only getting better.

[alexcrichton]

@sujayakar's done some awesome work to prototype this on OSX at least, discovering:

• For now, codegen-units probably has to be one to make this work (worth investigating)
• The pass to insert into LLVM is -C passes=insert-gcov-profiling
• The runtime support is available through -C link-args=-lclang_rt.profile_osx, and that library
• This runtime support lives around the directory xcrun --find clang in a path like usr/lib/clang/7.3.0/lib/darwin/libclang_rt.profile_osx.a.
• Binaries run with that instrumentation will produce a .gcda file
• Next to xcrun --find clang, there's an llvm-cov tool, and that can be run over the output file to generate information.

That should at least get code coverage working on OSX in some respect! THis is also a pretty strong case that it may not be too hard to actually get this working for all platforms if LLVM's standard tool suite "just works".

[sanxiyn]

@alexcrichton That's gcov coverage, which is completely different from what @gnzlbg meant in this issue.

The correct pass name is instrprof, implemented in lib/Transform/Instrumentation/InstrProfiling.cpp. See also LDC LLVM profiling instrumentation.

[gnzlbg]

@sanxiyn is correct.

I just want to add that I don't think we should implement clang-like gcov coverage generation.

LLVM format can be used for many more things (e.g. profile guided optimizations), and LLVM's llvm-cov tool can generate gcov files from it.

[frewsxcv]

afaik, there's a PR open for this: #38608

[sanxiyn]

As far as I can tell, #38608 is still gcov coverage and not instrprof.

[alexcrichton]

I believe this is now implemented as -Z profile and tracked at #42524, so closing in favor of that.

[sanxiyn]

This is not implemented yet. LLVM coverage is different from gcov coverage, which is also different from sanitizer coverage. LLVM implements three separate coverage mechanisms.

[mehcode]

Would LLVM coverage ( instrprof ) enable usable code coverage of generic heavy code? None of the existing code coverage solutions work well with heavy use of generics.

My project (a gameboy emulator) makes very heavy usage of generics and it'd be wonderful to get code coverage working on integration tests but I'm not really sure how.

[gnzlbg]

I think good code coverage support is the single most important piece of tooling missing in Rust.

When I run cargo test, and everything is green, I get a good feeling, but this feeling actually means nothing at all. To make cargo test mean something, it would need to at least tell me how many code-paths of my application have been exercised.

This information does not mean much either (just because a code-path was exercised does not mean that it was exercised for all inputs), but it would mean more than nothing, and it would make writing tests and asserting the value of test way easier.

IMO adding this kind of capability to cargo test (and yes, it should be part of cargo test, people should always know their code coverage) would be extremely valuable.

There is only another part of tooling infrastructure that would come close to this in value, and that would be an undefined behavior detector for all rust code.

This might sound like a rant, but I just want to raise awareness that this is a very important issue at least for me (and from the other issues being filled, for others as well) because it directly affects the correctness of rust programs (we don't want them to only be memory safe, but also to have correct logic).

Maybe if rustfmt, clippy, and the RLS advance enough during the impl period, the Rust tool team could prioritize these two tools next?

If I don't have precise auto-completion information or my source code is not perfectly formatted, well, I can live with that. But if my program has undefined behavior and I am not catching it because I am only testing 40% of my code-paths then... I am screwed. Does this make sense?

[mssun]

This is not implemented yet. LLVM coverage is different from gcov coverage, which is also different from sanitizer coverage. LLVM implements three separate coverage mechanisms.

Hi, can anyone explain that what's the difference between "LLVM coverage" (profile-instr-generate) and "gcov" (insert-gcov-profiling)? I don't know the detailed implementations of these tow passes, but will profile-instr-generate be more accurate than insert-gcov-profiling.

[mssun]

I found @kennytm's answer to my question. Let me put here in case someone like me gets lost in the overwhelmed references: #44673 (comment)

[bossmc]

I've done a bit of prototyping on this to see what would be involved in using instrprof for coverage, and it's revealed (to me) that the coverage aspect of this feature is secondary to the "Profile Guided Optimization" feature that instrprof was originally written for.

So, here's a vague "plan of action" (with progress so far) to enable PGO and LLVM-assisted coverage:

• Add frontend-guided instrumentation to IR
  • Unstable
• Enhance the profile-rt library to dump this out
• Enable Profile-Guided Optimization passes
  • Unstable
• Create a coverage map
• Integrate LLVM coverage tooling into cargo

Add Frontend-Guided Instrumentation

This is currently implemented in its most basic form as -Zpgo-gen which adds the -pgo-instr-gen and instrprof passes to LLVM to add instrumentation of all functions and basic blocks (loop bodies, if statements etc.).

There may be other places that it's useful to add instrumentation (e.g. adding vtable-downcast counts to enable -pgo-icall-prom passes - which turn dynamic dispatch into static dispatch in fast paths).

👍

Enhance the profile-rt library

With the above work complete, the generated binaries will count (in-memory) each time an instrumented code path is hit. To extract this information, the profile-rt library should walk the various __llvm_pfr_X sections of the binary/memory and dump the values out to a file (Clang dumps to default.profraw). The format of this file is (as far as I can see) "documented" in LLVM/Clang's codebase. On the other hand, the current implementation of Rustc's profile-rt library is taken wholesale from LLVM and thus already supports this function.

👍

Enable profile-guided optimization

With a default.profraw, one can create a profdata file (using llvm-profdata merge). This file can be passed to the pgo-instr-use (and other) passes to enable optimisations based on gathered profile information. Again, this is supported in nightly Rustc today (through -Zpgo-use).

👍

Create a coverage map

LLVM's coverage tooling can use a "Coverage Map" to map between instrumentation counters and blocks of code. The format of the map is well documented at http://llvm.org/docs/CoverageMappingFormat.html and there's code in LLVM (http://llvm.org/doxygen/classllvm_1_1coverage_1_1CoverageMappingWriter.html) for creating the magic format. Note that this may require Rustc manually inserting instrumentation (rather than using -pgo-instr-gen) since it needs to know which counter corresponds to which code block. The code in clang (https://github.com/llvm-mirror/clang/blob/master/lib/CodeGen/CodeGenPGO.cpp) that does this is probably a good place to start.

Note that the LLVM coverage map format supports various useful features, not well handled in GCOV-style coverage:

• Column as well as line-number analysis (useful for inline expressions)
• Macro-expansion analysis (coverage can "see through" macros)

❌

Integrate coverage tooling into Cargo

Creating a cargo coverage (or, possibly better, just creating coverage from cargo test) by adding profiling to the test cargo-profile and running the resulting profile data through llvm's tooling would be really cool. Especially if the output format is compatible with tools like Coveralls.io and can produce useful graphical outputs for analysis afterwards.

Likely requires stabilising -Zpgo-gen or at least exposing enough of it that cargo can create a profiled build.

❌

---

Unanswered Questions

There are some bits of Rust code for which coverage is non-obvious:

• Do landing pads report coverage? If they do then we run into the same issue as Rustc adds line-number information for unhittable panic handlers #55352
  • Maybe we add instrumentation for them (to determine how often they're hit) but don't add those counters to the coverage map?
• How do we handle non-instantiated generics? If a generic is never instantiated then no code is created for it so there's no counters for it.
  • Presumably we need to ensure that the coverage map includes the counters that "would be there if there was an instantiation" then coverage will spot that the generic is not covered.
  • This applies to "default implementations" in traits and likely is impacted by specialisation too.
• Similarly, should separate instantiations of the same generic be covered separately? If I test Foo<i32> in my UTs but use Foo<String> in my real code, have I really tested my code (in this case, String needs drop so maybe there's a subtlety there)?
  • Create coverage map entries for all instantiations (as well as the "generic" counter entry from the previous bullet)? I don't think llvm-cov knows how to handle this in the coverage map at the moment, but it might make sense for it to do so to support this for C++ templates too?

[bossmc]

• Proc-macros (and the built-in #derives) need care too - the expanded code coverage is potentially uninteresting (if you trust the proc-macro) or critical (if you don't).
• Basically anywhere where rustc does "codegen" above and beyond what the user wrote needs to be thought about quite carefully.

[pJunger]

* How do we handle non-instantiated generics?  If a generic is never instantiated then no code is created for it so there's no counters for it.

I think rust should behave the same as clang here.
And if I remember correctly, then not instantiated templates are reported.

* Similarly, should separate instantiations of the same generic be covered separately?  If I test `Foo<i32>` in my UTs but use `Foo<String>` in my real code, have I really tested my code (in this case, String needs `drop` so maybe there's a subtlety there)?

llvm-cov at least has the option to show coverage per instantiation (or merged - per -show-instantiations).
So I think it makes sense to provide this as well.

[eggyal]

Great write-up @bossmc! I'll take a stab at adding LLVM coverage map generation, unless it's already being worked on elsewhere?

Whilst I'm happy to try and figure out a solution on my own, a little direction from the compiler team would no doubt ensure that I arrive at something that's acceptable far sooner! Would anyone be willing to mentor this?

Cc @michaelwoerister

[michaelwoerister]

I don't have time to mentor this but feel free to ask specific questions!

[eggyal]

@bossmc Are you sure that -Zpgo-gen (now stabilised as -C profile-generate) adds LLVM's -pgo-instr-gen and -instrprof passes? My understanding is that it "merely" instruments via gcov, which is the very issue under discussion here? Have I missed something?

[BenChand]

@richkadel

Please cc me on any bug reports.

Hi,

I've been encountering a problem with generating code coverage for a project that uses the crate prometheus, specifically versions 0.9 onwards. Using these versions of the prometheus crate results in the error Failed to load coverage: Truncated coverage data when attempting to build the final coverage report. I've attempted to generalize the problem into a small test project here:

https://github.com/BenChand/prometheus-code-cov

I'm currently using nightly-2020-11-25. I've also listed 3 ways that I am able to get code coverage working in that example project when I change parts related to the prometheus crate.

I've tried with these versions of LLVM:

• 11.0.0-rust-1.50.0-nightly
• 11.0.1

Thanks

[richkadel]

@BenChand - Thanks for putting this together! I'm taking a closer look at recent bug reports on coverage this week.

For tracking purposes, can you please copy this info into a Rust bug report:

https://github.com/rust-lang/rust/issues/new?labels=C-bug&template=bug_report.md

Thanks!

[frazerk]

Hello all,

Thanks for this! I've been experimenting with it for a few days, and it's working well. I've had a couple of issues, however, and I figured it was best to mention them here, just in case I should file bugs or feature requests.

First, is there any way to exclude the tests themselves from the coverage report? In tests that give a message to assert! macros (e.g. assert_eq!(sut(), "test", "must return test");), the region containing the message is never executed in a successful test and shows up in the report as an uncovered region. I'm guessing this is functionality that would need to be added to llvm-cov.

Second, I'm having trouble specifying the locations of source files. Commands like llvm-cov show do work, even if no source files are given as command-line arguments, but some options (like llvm report --show-functions) require them. However, regardless of whether I specify source files as absolute paths or relative, llvm-cov can never find them:

$ llvm-cov report -Xdemangler=rustfilt -instr-profile=target/default.profdata --ignore-filename-regex=.cargo|/rustc|.rustup --show-functions $binary src/main.rs
error: Source files must be specified when -show-functions=true is specified

I'm not totally certain what the problem is, but it looks like crate source files are being put into the coverage reporting as relative paths, but llvm-cov can only match source files by absolute path. If you look at the output of llvm-cov report (without using --ignore-filename-regex), all of the dependencies are mentioned by absolute path, but crate sources are relative:

$ llvm-cov report -Xdemangler=rustfilt -instr-profile=target/default.profdata $binary
Filename                                                                                                                                Regions    Missed Regions     Cover   Functions  Missed Functions  Executed       Lines      Missed Lines     Cover
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
/home/user/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/anyhow-1.0.35/src/chain.rs                                                        25                25     0.00%           3                 3     0.00%          32                32     0.00%
/home/user/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/anyhow-1.0.35/src/error.rs                                                         9                 9     0.00%           3                 3     0.00%          14                14     0.00%
/home/user/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/anyhow-1.0.35/src/fmt.rs                                                          80                80     0.00%           3                 3     0.00%          55                55     0.00%
/home/user/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/anyhow-1.0.35/src/kind.rs                                                          3                 3     0.00%           1                 1     0.00%           3                 3     0.00%
<snip>
src/main.rs                                                                                                                                   1                 0   100.00%           1                 0   100.00%           1                 0   100.00%
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                                     31604             30367     3.91%        3590              3217    10.39%       35581             33247     6.56%

If you run with the undocumented option --dump-collected-paths, source files that were specified as relative paths will still be printed as absolute paths:

$ llvm-cov report -Xdemangler=rustfilt -instr-profile=target/default.profdata $binary --dump-collected-paths src
/home/user/code/test_crate/src/main.rs

I'm not certain this is what's causing the problem, but it is suspicious.

Overall, though, everything is working well! Thanks so much for all the hard work on this.

In case it's useful, here's the small script I've been using to generate coverage in everyday use:

#!/bin/bash
IGNORE=".cargo|/rustc|.rustup"

set -euxo pipefail
export RUSTFLAGS="-Zinstrument-coverage"
rm -f target/default.profdata target/default.profdata

crate_name=$(cargo read-manifest | jq -r '.name')
test_binary=$(cargo build --tests --message-format json |
        jq -r "select(.reason==\"compiler-artifact\") | select(.target.name == \"${crate_name}\") | .filenames | .[0]")

LLVM_PROFILE_FILE=target/default.profraw cargo test -v
llvm-profdata merge -sparse target/default.profraw -o target/default.profdata

subcommand=report
options=

if [[ $# -gt 0 && $1 == "show" ]]; then
    subcommand=show
    options='--show-instantiations --show-line-counts-or-regions'
fi

llvm-cov $subcommand \
    -Xdemangler=rustfilt \
    -instr-profile=target/default.profdata \
    --ignore-filename-regex="$IGNORE" \
    $options \
    "$test_binary"

[catenacyber]

@frazerk To get absolute paths for every file, until rust-lang/cargo#5450 or similar is merged, I use --remap-path-prefix in $RUSTFLAGS
cf https://github.com/google/oss-fuzz/pull/4697/files#diff-fe54775f01d65c2afafd711ef7e68c31192a6f8a8d71dd8e06c62ef25a32e400R26

llvm-cov also accepts some option -path-equivalence which might help you

[richkadel]

FYI, for contributors, or anyone interested in the implementation details, I added a section on LLVM Source-Based Code Coverage to the Rustc Dev Guide.

[briansmith]

$ llvm-cov report -Xdemangler=rustfilt -instr-profile=target/default.profdata --ignore-filename-regex=.cargo|/rustc|.rustup --show-functions $binary src/main.rs
error: Source files must be specified when -show-functions=true is specified

Try:

llvm-cov report -Xdemangler=rustfilt -instr-profile=target/default.profdata --ignore-filename-regex=.cargo|/rustc|.rustup --show-functions $binary src/main.rs .

(Note the trailing dot.)

[briansmith]

(Note the trailing dot.)

To expand on it a little more, it seems if you pass a directory as one (or more?) of the SOURCES on the command line, llvm-cov will resolve relative paths using that directory as a base.

[frazerk]

I use --remap-path-prefix in $RUSTFLAGS

This worked! Adding --remap-path-prefix src=${PWD}/src to $RUSTFLAGS was enough to fix llvm-cov report -show-functions. Also, since llvm-cov can now actually find source files, I don't need to use the -ignore-filename-regex option to remove dependencies anymore, since it will filter down to the src directory when given as an argument.

(Note the trailing dot.)

This didn't seem to make a difference, unfortunately. I'm pretty sure llvm-cov resolves all source files to absolute paths, regardless of how they're specified.

Now the only thing I'm missing is a way to exclude the tests themselves from coverage reports. It looks like llvm-cov's -name-regex option for controlling which functions show up in the output only applies to the show subcommand. Is this something I should file a feature request for? If so, is this something that should be solved rust-side (i.e. not instrumenting functions in a #[cfg(test)] module) or llvm-side (i.e. better filtering in llvm-cov)?

[richkadel]

Thanks for highlighting these use cases and solutions @frazerk, @briansmith, and @catenacyber !

@frazerk - This sounds like a possible limitation in llvm-cov, and if so, I think the issue/feature request should be filed on the LLVM side.

Thanks!

[pJunger]

Thank you very much for this implementation! I've got it to work & the results look really good!
I have a question regarding coverage of derived traits: What exactly does it mean that a trait implementation is not covered?
Are any/all of those derived functions uncovered? Regions inside those derived functions?

[jokeyrhyme]

It might mean that the traits aren't sufficiently instrumented, or it could mean that none of the executed code (e.g. your tests) actually exercise those traits

[richkadel]

none of the executed code (e.g. your tests) actually exercise those traits

That seems to be what it means, from what I've observed.

[pJunger]

Sorry, I think I've been unclear: I agree that in the linked cases the code really has not been executed at all, and I expected them to be not covered.
But the question (to me at least) remains:
When is a derived implementation counted as covered? Is it covered if any of the generated functions is called (that wouldn't be that helpful if a trait has multiple functions) or would it still count as uncovered if e.g. PartialEq's eq method has been called, but ne hasn't?

[richkadel]

I'm fairly sure it counts coverage for each function executed in the derived trait, but I don't know how llvm-cov show would be able to visualize distinct functions. For example, If eq and ne are executed, I believe it will show ^2, but if only eq is executed, twice, I think it would also show ^2. llvm-cov show can only overlay counts over the original source.

Note, there is an open issue related to how the counters are displayed for derived traits (#79626), but that issue isn't attempting to address how to show counters for individual functions from the derived trait. Just something to be aware of. If you have a suggestion for how this could be done, please feel free to post a feature request.

[xd009642]

Just a few questions that occurred to me:

1. Is there a way to rename the default.profraw and also choose a location to save it? (might be nice to go in the target dir)
2. If a test launches another process in the project (e.g. for CLI integration tests) and will spawning a new process that's instrumented cause a new default.profraw to be created conflicting with the other one? I imagine one will just delete the other

[pJunger]

Thank you for your answer! I don't have any suggestion on how to get the derived traits displayed - if I'm not mistaken those macros would need to be expanded by rustc(?) & I doubt that the LLVM project would want that included in llvm-cov.

@xd009642 Regarding the first - it's already mentioned in the nightly book & using the target directory works as output directory does work.
And at least Clang deletes the old profraw file - I would expect the same here. That's why I don't understand that the special patterns for the file name do not include something like a timestamp.

[richkadel]

@pJunger I think I have to correct my reply here. While experimenting with an unrelated issue, I happened to compile a test program that executed eq and ne, and I was pleasantly surprised to see that these functions are counted separately:

   23|      4|//! #[derive(Debug, PartialEq)]
  ------------------
  | <rust_out::main::_doctest_main_src_test_run_make_fulldeps_coverage_doctest_rs_22_0::SomeError as core::cmp::PartialEq>::ne:
  |   23|      3|//! #[derive(Debug, PartialEq)]
  ------------------
  | <rust_out::main::_doctest_main_src_test_run_make_fulldeps_coverage_doctest_rs_22_0::SomeError as core::cmp::PartialEq>::eq:
  |   23|      1|//! #[derive(Debug, PartialEq)]
  ------------------

llvm-cov show views the coverage of each trait function as a different variant for the same code region, in the same way generics and closures are handled.

However, if only one of the functions is executed, I don't think it will be obvious which function it was.

[maboesanman]

Would it be possible to exclude lines which call unreachable!() ?

for example, here is a file from the coverage for my project:

note that unreachable!() is marked as missed despite this being correctly covered.

[richkadel]

@maboesanman That link doesn't work for me. (I see you replaced the link with an image, thanks.)

All - It would be better to file separate Issues or feature requests that can be tracked, rather than adding additional requests to this Issue. (I think this issue is resolved and should be closed?)

[maboesanman]

@richkadel where do you suggest these feature requests get made?

[pJunger]

@richkadel Thank you for the update! I had hoped that the export command would provide more information on this.
But it seems like uncovered functions don't show up in the exported JSON as well. Maybe some future version of llvm-cov will provide better information.

[richkadel]

@maboesanman - I'm not an expert on the rust-lang/rust processes, but starting with https://github.com/rust-lang/rust/issues/new/choose, I see there's an entry for "Feature Requests", and granted, it doesn't suggest filling out a form, but there's a link and a note, "Please discuss language feature requests on the internals forum."

Perhaps someone from the compiler team can provide better instructions on this if needed.

But the main point is, if there are feature requests, they should be new issues, so this one can be cleanly closed out.

Thanks for understanding!

[richkadel]

@xd009642 - also, you may be interested in the discussion at #79899, and the suggestion by @vmx to have cargo set default LLVM_PROFILE_FILE paths/names. I'm not sure if @vmx is thinking about implementing this, or if someone else is interested. (I don't have the time myself, due to shifting work priorities.) But I like the idea in general.

[wesleywiser]

I think this issue can now be closed. @richkadel has done a great job implementing this under the unstable -Zinstrument-coverage flag. Stabilization for this flag is being tracked in #79121 and related enhancements and bugs can be found by looking at the A-code-coverage label.

Thanks @richkadel, @tmandry and everyone else for your help on implementing this feature! 🎉