Source-based coverage doesn't support branch coverage #520
Comments
|
Support for branch coverage in LLVM and Clang is hopefully going to be added soon, https://reviews.llvm.org/D84467. I think it will need porting to Rust before it can be used. In the meantime, it might be possible to extrapolate branch coverage from region coverage, even though branch coverage would offer additional information (e.g. how often a given subcondition in an expression was hit) that region coverage can't offer by itself. |
rivy
added a commit
to rivy/rs.platform-info
that referenced
this issue
Feb 13, 2023
## [why] Coverage calculations using the past `grcov` method has become increasingly ... odd, with strange lapses not obviously related to code changes. More recently, Rust (and `grcov`) have been promoting a new method for code coverage calculation, called instrumented (or source-based). Discussion about instrumented coverage has now been stabilized and included in [_The rustc book_][^1]. Unfortunately, the current incarnation of instrumented coverage does not support branch coverage calculation. It's, at best, a work-in-progress, possibly a works-as-designed, with, currently, a lack of strong support for adding branch support. "Region" coverage calculation is instead being discussed as a substitute for branch calculations. Ultimately, given increasing flakiness with the current method, a more robust and accurate method of line coverage is worth the loss of information arising from lack of branch coverage, which is of lower importance for most development. ### refs [^1]: https://doc.rust-lang.org/rustc/instrument-coverage.html - [Rust ~ Instrumentation-based coverage](https://doc.rust-lang.org/rustc/instrument-coverage.html) @@ <https://archive.is/ERWrk> - [HowTo collect Rust source-based coverage](https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html) @@ <https://archive.is/X9R14> - [`grcov` issue ~ missing branch coverage](mozilla/grcov#520) - [Rust issue ~ add support for branch coverage](rust-lang/rust#79649)
rivy
added a commit
to rivy/rs.platform-info
that referenced
this issue
Feb 14, 2023
## [why] Coverage calculations using the past `grcov` method has become increasingly ... odd, with strange lapses not obviously related to code changes. More recently, Rust (and `grcov`) have been promoting a new method for code coverage calculation, called instrumented (or source-based). Discussion about instrumented coverage has now been stabilized and included in [_The rustc book_][^1]. Unfortunately, the current incarnation of instrumented coverage does not support branch coverage calculation. It's, at best, a work-in-progress, possibly a works-as-designed, with, currently, a lack of strong support for adding branch support. "Region" coverage calculation is instead being discussed as a substitute for branch calculations. Ultimately, given increasing flakiness with the current method, a more robust and accurate method of line coverage is worth the loss of information arising from lack of branch coverage, which is of lower importance for most development. ### refs [^1]: https://doc.rust-lang.org/rustc/instrument-coverage.html - [Rust ~ Instrumentation-based coverage](https://doc.rust-lang.org/rustc/instrument-coverage.html) @@ <https://archive.is/ERWrk> - [HowTo collect Rust source-based coverage](https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html) @@ <https://archive.is/X9R14> - [`grcov` issue ~ missing branch coverage](mozilla/grcov#520) - [Rust issue ~ add support for branch coverage](rust-lang/rust#79649)
rivy
added a commit
to rivy/rs.platform-info
that referenced
this issue
Feb 14, 2023
## [why] Coverage calculations using the past `grcov` method has become increasingly ... odd, with strange lapses not obviously related to code changes. More recently, Rust (and `grcov`) have been promoting a new method for code coverage calculation, called instrumented (or source-based). Discussion about instrumented coverage has now been stabilized and included in [_The rustc book_][^1]. Unfortunately, the current incarnation of instrumented coverage does not support branch coverage calculation. It's, at best, a work-in-progress, possibly a works-as-designed, with, currently, a lack of strong support for adding branch support. "Region" coverage calculation is instead being discussed as a substitute for branch calculations. Ultimately, given increasing flakiness with the current method, a more robust and accurate method of line coverage is worth the loss of information arising from lack of branch coverage, which is of lower importance for most development. ### refs [^1]: https://doc.rust-lang.org/rustc/instrument-coverage.html - [Rust ~ Instrumentation-based coverage](https://doc.rust-lang.org/rustc/instrument-coverage.html) @@ <https://archive.is/ERWrk> - [HowTo collect Rust source-based coverage](https://marco-c.github.io/2020/11/24/rust-source-based-code-coverage.html) @@ <https://archive.is/X9R14> - [`grcov` issue ~ missing branch coverage](mozilla/grcov#520) - [Rust issue ~ add support for branch coverage](rust-lang/rust#79649)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi there, nice job with grcov!
We've been having some issues with coverage for our project (probably not grcov's fault), and when I was working on fixing it I discovered the new source-based coverage, which looked like the way forward (given you use it youself in your CI). So I adopted it, but noticed branch coverage was 0/0. (the PR in question: ActivityWatch/aw-server-rust#192)
Is this planned/intended? From what I've read gcov's source-based coverage supports branches just fine. (https://clang.llvm.org/docs/SourceBasedCodeCoverage.html)
Thanks for your work! ❤️
Edit: After reading the rust unstable book on source-based code coverage it seems "branch coverage" is superseded by "region coverage". Is this a correct interpretation? Is "branch coverage" redundant in a "region coverage" paradigm?
The text was updated successfully, but these errors were encountered: