Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No C++ coverage collected when a runtime object is not in runfiles #15121

Closed
fmeum opened this issue Mar 25, 2022 · 0 comments
Closed

No C++ coverage collected when a runtime object is not in runfiles #15121

fmeum opened this issue Mar 25, 2022 · 0 comments
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) team-Rules-CPP Issues for C++ rules type: bug

Comments

@fmeum
Copy link
Collaborator

fmeum commented Mar 25, 2022

Description of the problem / feature request:

When a cc_test transitively depends on a cc_binary with linkshared = True, the latter is added to the list of runtime objects to collect coverage for. However, if e.g. the dependency on the shared library goes through a java_binary's resource attribute, the shared library will not be available in the cc_test's runfiles, which causes LLVM-backed coverage collection to not produce any C++ coverage reports for the entire test.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

$ touch WORKSPACE
$ cat << EOF > BUILD
cc_test(
    name = "main",
    srcs = ["main.cpp"],
    data = [":jar"],
)

java_binary(
    name = "jar",
    resources = [":shared_lib"],
    create_executable = False,
)

cc_binary(
    name = "shared_lib",
    linkshared = True,
)
EOF

$ cat << EOF > main.cpp
#include <iostream>

int main(int argc, char const *argv[])
{
  if (argc < 5) {
    std::cout << "Hello World!" << std::endl;
  }
}
EOF

$ cat << EOF > .bazelrc
# Standard setup of llvm-cov coverage.
coverage --repo_env=CC=clang
coverage --repo_env=BAZEL_USE_LLVM_NATIVE_COVERAGE=1
coverage --repo_env=GCOV=llvm-profdata
coverage --combined_report=lcov
coverage --experimental_use_llvm_covmap
coverage --experimental_generate_llvm_lcov
# Simplifies the reproducer.
coverage --instrument_test_targets
EOF

$ bazel coverage //:main
...
WARNING: There was no coverage found.
INFO: LCOV coverage report is located at /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/execroot/__main__/bazel-out/_coverage/_coverage_report.dat
 and execpath is bazel-out/_coverage/_coverage_report.dat
Target //:main up-to-date:
  bazel-bin/main
INFO: Elapsed time: 0.769s, Critical Path: 0.59s
INFO: 2 processes: 2 linux-sandbox.
INFO: Build completed successfully, 2 total actions
//:main                                                                  PASSED in 0.4s

Executed 1 out of 1 test: 1 test passes.
INFO: Build completed successfully, 2 total actions

$ cat bazel-out/_coverage/_coverage_report.dat
<empty file>

$ bazel coverage //:main --test_env=VERBOSE_COVERAGE=1 --test_output=streamed -t-
...
+ /usr/bin/llvm-cov export -instr-profile /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/_coverage/main/test/_cc_coverage.dat.data -format=lcov '-ignore-filename-regex=.*external/.+' -ignore-filename-regex=/tmp/.+ -object /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/main -object /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/libshared_lib.so -object /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/libshared_lib.so
+ sed s#/proc/self/cwd/##
error: /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/main, /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/libshared_lib.so, /home/fhenneke/.cache/bazel/_bazel_fhenneke/7e105330b418afcd95502d6a63e09121/sandbox/linux-sandbox/29/execroot/__main__/bazel-out/k8-fastbuild/bin/main.runfiles/__main__/libshared_lib.so: Failed to load coverage: No such file or directory
...

Repeating the commands without the data dependency on :jar shows that coverage is correctly collected in that case.

What operating system are you running Bazel on?

Linux

What's the output of bazel info release?

5.1.0, but also applies to current HEAD

Have you found anything relevant by searching the web?

I created a potential fix at #15118.

Any other information, logs, or outputs that you want to share?

While this particular example is synthetic, the scenario in which this issue causes coverage collection to fail is common: JNI libraries, perhaps even built for multiple architectures, are often wrapped in jars and extracted at runtime (AFAIK, Bazel itself does this). The same problem also applies to other situations, e.g. when an artifact is copied to a different location by an intermediate rule and only included in the runfiles at that new location.

@ckolli5 ckolli5 added team-Rules-CPP Issues for C++ rules untriaged labels Mar 25, 2022
@oquenchil oquenchil added type: bug P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) and removed untriaged labels Mar 28, 2022
@oquenchil oquenchil removed their assignment Mar 28, 2022
fmeum added a commit to fmeum/bazel that referenced this issue Apr 20, 2022
Before this commit, collecting C++ coverage in lcov format would fail
at the llvm-cov export step if a shared library listed in the
runtime_objects_list.txt was not contained in the runfiles of the top-
level target. This can happen e.g. if a cc_library depends on a
java_binary that has a cc_binary shared library in its resources.

This is fixed by not including objects that don't exist at runtime
in the llvm-cov invocation.

Fixes bazelbuild#15121.

Closes bazelbuild#15118.

PiperOrigin-RevId: 442799461
ckolli5 pushed a commit that referenced this issue May 9, 2022
Before this commit, collecting C++ coverage in lcov format would fail
at the llvm-cov export step if a shared library listed in the
runtime_objects_list.txt was not contained in the runfiles of the top-
level target. This can happen e.g. if a cc_library depends on a
java_binary that has a cc_binary shared library in its resources.

This is fixed by not including objects that don't exist at runtime
in the llvm-cov invocation.

Fixes #15121.

Closes #15118.

PiperOrigin-RevId: 442799461
fmeum added a commit to fmeum/bazel that referenced this issue Jun 11, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 then
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
fmeum added a commit to fmeum/bazel that referenced this issue Jun 11, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 then
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
fmeum added a commit to fmeum/bazel that referenced this issue Jun 11, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 then
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
fmeum added a commit to fmeum/bazel that referenced this issue Jun 11, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 then
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
fmeum added a commit to fmeum/bazel that referenced this issue Jun 11, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 then
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
fmeum added a commit to fmeum/bazel that referenced this issue Jun 12, 2023
`--experimental_split_coverage_postprocessing` now works in combination
with `--experimental_generate_llvm_lcov`, which required adding the
runtime objects as inputs to the separate coverage post-processing
spawn. Previously, they were only added to the runfiles of the test
action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 than
bb6f1a7, which added an additional
check to only add those runtime objects to the `llvm-cov` command line
that were present in runfiles. Now, since all runtime objects are staged
as inputs, they are known to be available. This improves coverage
accuracy when e.g. runtime objects are packaged into jars and loaded at
runtime.
copybara-service bot pushed a commit that referenced this issue Jun 20, 2023
`--experimental_split_coverage_postprocessing` now works in combination with `--experimental_generate_llvm_lcov`, which required adding the runtime objects as inputs to the separate coverage post-processing spawn. Previously, they were only added to the runfiles of the test action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for #15121 than bb6f1a7, which added an additional check to only add those runtime objects to the `llvm-cov` command line that were present in runfiles. Now, since all runtime objects are staged as inputs, they are known to be available. This improves coverage accuracy when e.g. runtime objects are packaged into jars and loaded at runtime.

Along the way the common setup for LLVM-based coverage tests has been extracted into a helper function and the tests have been updated to work with a broader range of clang versions. Previously, tests weren't run at all due to the clang version on the runners not falling into the restrictive range. Now, the tests are executed on the Ubuntu 20.04 runner (but not on any other runners, including macOS).

Closes #18634.

PiperOrigin-RevId: 542055078
Change-Id: Id8851f8685bb7724891acca395bb91556ee2b29a
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Jun 21, 2023
`--experimental_split_coverage_postprocessing` now works in combination with `--experimental_generate_llvm_lcov`, which required adding the runtime objects as inputs to the separate coverage post-processing spawn. Previously, they were only added to the runfiles of the test action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for bazelbuild#15121 than bb6f1a7, which added an additional check to only add those runtime objects to the `llvm-cov` command line that were present in runfiles. Now, since all runtime objects are staged as inputs, they are known to be available. This improves coverage accuracy when e.g. runtime objects are packaged into jars and loaded at runtime.

Along the way the common setup for LLVM-based coverage tests has been extracted into a helper function and the tests have been updated to work with a broader range of clang versions. Previously, tests weren't run at all due to the clang version on the runners not falling into the restrictive range. Now, the tests are executed on the Ubuntu 20.04 runner (but not on any other runners, including macOS).

Closes bazelbuild#18634.

PiperOrigin-RevId: 542055078
Change-Id: Id8851f8685bb7724891acca395bb91556ee2b29a
iancha1992 added a commit that referenced this issue Jun 21, 2023
`--experimental_split_coverage_postprocessing` now works in combination with `--experimental_generate_llvm_lcov`, which required adding the runtime objects as inputs to the separate coverage post-processing spawn. Previously, they were only added to the runfiles of the test action, which aren't (and shouldn't) be added to the coverage spawn.

As a side effect, this provides a more natural fix for #15121 than bb6f1a7, which added an additional check to only add those runtime objects to the `llvm-cov` command line that were present in runfiles. Now, since all runtime objects are staged as inputs, they are known to be available. This improves coverage accuracy when e.g. runtime objects are packaged into jars and loaded at runtime.

Along the way the common setup for LLVM-based coverage tests has been extracted into a helper function and the tests have been updated to work with a broader range of clang versions. Previously, tests weren't run at all due to the clang version on the runners not falling into the restrictive range. Now, the tests are executed on the Ubuntu 20.04 runner (but not on any other runners, including macOS).

Closes #18634.

PiperOrigin-RevId: 542055078
Change-Id: Id8851f8685bb7724891acca395bb91556ee2b29a

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P4 This is either out of scope or we don't have bandwidth to review a PR. (No assignee) team-Rules-CPP Issues for C++ rules type: bug
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants