Python Coverage does not work

[ulfjack]

I thought that there was an existing issue for this, but I can't find it.

I was able to successfully collect coverage for python code with a modified version of coverage.py:

1. Use a Bazel binary that include b01c859
2. Check out the modified coverage.py from https://github.com/ulfjack/coveragepy/tree/lcov-support (this is based on the patch at DO-NOT-MERGE: Patched-on LCOV file format support for coveragepy v4.x nedbat/coveragepy#863 with a small change to make it work with Bazel)
3. Run bazel:

bazel coverage --test_env=PYTHON_COVERAGE=/path/to/coveragepy/coverage/__main__.py //python:lib_test

Test coverage should be written to bazel-testlogs/python/lib_test/coverage.dat in lcov format.

[ulfjack]

In order for cross-language coverage to work in Bazel, all coverage data needs to be converted into a common format. The above workaround uses an upstream patch to coverage.py. As an alternative, we could also change LcovMerger to support coverage.py's output format.

Reference to upstream's issue: nedbat/coveragepy#587

[dmadisetti]

The issue you were looking for may have been under python_rules bazelbuild/rules_python#43

Thanks for the suggested workaround!

[zuerst]

@ulfjack thank you for the suggestion. Unfortunately I am seeing empty coverage.dat file with some warnings in in the test.log file. Any idea what I might be missing?

test.log

Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getTracefiles
INFO: No lcov file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getGcovInfoFiles
INFO: No gcov info file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main getProfdataFileOrNull
INFO: No .profdata file found.
Jul 08, 2020 6:35:19 PM com.google.devtools.coverageoutputgenerator.Main main
WARNING: There was no coverage found.

Also is there a documentation about using Bazel Coverage on python that might be helpful to read?

[ulfjack]

The most likely reason for coverage not to work is an incorrect --instrumentation_filter, and the second most likely is having sources in a *_test rule - test rules do not collect any coverage. Beyond that, there are a number of things that could go wrong, but I don't know which one it is. You'd have to post a repro if you need more help.

[chickenandpork]

@ulfjack I get empty coverage files regardless what I try, and these are repositories with only __init__.py and test_...py files in the srcs attribute. The code under test in one project are in libraries and referenced as dependencies (ie not in srcs, but also not an embed of any kind).

Do you have a basic example that you used during development? Or a test case that works? It would be really helpful to see any sort of functioning test to fast-track some experimentation.

For reference, I've tried various combinations of this script, changing the parameters of the last line to get a run that doesn't fail, but gives non-zero coverage.dat files. So far, never ever a nonzero coverage.dat.

#!bash

COVERAGEPY=${HOME}/src/coveragepy
test -d "${COVERAGEPY}" || git clone   -b lcov-support    https://github.com/ulfjack/coveragepy.git "${COVERAGEPY}"

# According to https://github.com/bazelbuild/bazel/issues/10660, and some assumption:

bazel coverage --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py  //:test_ns

# Now, in other documentation, there is much discussion about local_jdk and various coverage flags.  
# The following completes without error, but still empty coverage.dat files
#
# NOTE: //:test_ns is the only test target for this example.

bazel coverage -s  --test_env=PYTHON_COVERAGE=${COVERAGEPY}/coverage/__main__.py  --sandbox_debug \
    --instrument_test_targets   --instrumentation_filter=//...:all --combined_report=lcov \
    --javabase=@bazel_tools//tools/jdk:remote_jdk11  //:test_ns

Sometimes I also try with --coverage_report_generator= @bazel_tools//tools/test/CoverageOutputGenerator/java/com/google/devtools/coverageoutputgenerator:Main but there seems no difference.

[dmadisetti]

@chickenandpork I've been generating reports for a while now. Not directly with bazel, but by manually calling grcov in conjunction with coveragepy. Here's a sample from one of my private repo actions:

    - name: Python coverage
      run: |
        cd "${GITHUB_WORKSPACE}/src"
        curl -L https://github.com/ulfjack/coveragepy/archive/lcov-support.tar.gz | tar xvz
        ~/bin/bazel coverage -t- --instrument_test_targets --experimental_cc_coverage \
            --test_output=errors --linkopt=--coverage --linkopt=-lc \
            --test_env=PYTHON_COVERAGE=${GITHUB_WORKSPACE}/src/coveragepy-lcov-support/__main__.py \
            --define=config_file=test //python/:tests
        # Unfortunately, coverage does not produce the hit information. Running
        # the compiled tests explicitly seems to work though.
        find bazel-bin/python/ -maxdepth 1 -regex ".*-test$" -exec {} \;
        cp bazel-out/k8-fastbuild/testlogs/pd3/python/smoke-test/coverage.dat coverage.py.info
      env:
        CC: clang-9

    - name: codecov
      if: always()
      run: |
        set -x
        cd "${GITHUB_WORKSPACE}/src"
        ~/bin/grcov coverage.py.info bazel-bin/python/_objs/ -t lcov \
          --ignore "external/*" --ignore "/usr/*" \
          --ignore "*deps_*" --ignore "*_pb2.py"  \
          --llvm > lcov.py.info
        sed -i 's/SF:.*test\.runfiles\/python\/python/SF:python/g' coverage.py.info
        cd ..
        bash -x <(curl -s https://codecov.io/bash) -Z -v -f src/lcov.py.info -F python_tests
      env:
        CC: clang-9
        CODECOV_TOKEN: ${{ secrets.CODE_COV }}

[lberki]

/cc @oquenchil

Maybe this can fit into your coverage sprint?

[vp3tra]

@chickenandpork In case this is still relevant for you, or maybe this will help someone:

bazel's --instrumentation_filter option is a regular expression, so passing --instrumentation_filter=//...:all will not have the effect of instrumenting everything, as one would expect. Instead likely it will not instrument anything.

Leaving the option unset, bazel will generally guess a good value, or if you want to instrument everything all the time, then --instrumentation_filter=// might do what you intended.

[joshua-cannon-techlabs]

@chickenandpork @ulfjack If you use a pytest_runner to run your pytest tests, it's a convenient place to put the lconv conversion code ;) That way you can still use standard tools (and not need to use the fork or special shell scripts)

The following requires pytest-cov and coverage-lconv

import contextlib
import os
import pathlib
import sys

import coverage
import coverage_lcov.converter
import pytest

@contextlib.contextmanager
def coverage_decorator():
    if os.getenv("COVERAGE", None) == "1":
        coverage_dir = pathlib.Path(os.getenv("COVERAGE_DIR"))
        coverage_file = coverage_dir / ".coverage"
        coverage_manifest = pathlib.Path(os.getenv("COVERAGE_MANIFEST"))
        coverage_sources = coverage_manifest.read_text().splitlines()
        # @TODO: Handle config stuff
        cov = coverage.Coverage(data_file=str(coverage_file), include=coverage_sources)
        cov.start()

    try:
        yield
    finally:
        if os.getenv("COVERAGE", None) == "1":
            cov.stop()
            cov.save()

            # @TODO: Handle config stuff
            coverage_lcov.converter.Converter(
                relative_path=True,
                config_file=False,
                data_file_path=str(coverage_file),
            ).create_lcov(os.getenv("COVERAGE_OUTPUT_FILE"))


if __name__ == "__main__":
    with coverage_decorator():
        sys.exit(pytest.main(sys.argv[1:]))

[joshua-cannon-techlabs]

The dropbox Python rules also seem to do something similar for coverage support: https://github.com/dropbox/dbx_build_tools/blob/fe5c9e668a9e6951970c0595089742d8a0247b8c/build_tools/py/pytest_plugins/codecoverage.py

[alokpr]

py_test for pybind module does not work either. I expected it to work since the module is a cc_binary and hence gcov/lcov should be sufficient. All bazel commands - coverage|test|run <py-test-target> - produce empty coverage report.

Interestingly running the py-test-target directly produces .gcda files as expected. So it seems bazel is simply ignoring the .gcda files for py_test targets? Has anyone seen this before and knows of a workaround?

[aignas]

FYI, I created bazelbuild/rules_python#977 to make bazel coverage work with the hermetic Python toolchain from @rules_python. I found some issues whilst doing the work and I'll copy them here for visibility. The issues were:

• I had to use coverage.py v6.5.0 because the latest version (7.0.4 has a types.py file in the package directory, which imports from Python's stdlib types [1]. Somehow the Python interpreter is thinking that the from types import FrameType is referring to the currently interpreted file and everything breaks. I would have expected the package to use absolute imports and only attempt to import from coverage.types if we use coverage.types and not just a plain types import.
• The multi_python_versions example cannot show coverage for the more complex tests that are using subprocess to spawn a different Python interpreter. I am wondering if this is related to the fact that we are including coverage.py via the toolchain and not through other mechanisms [2].
• The __init__.py files in the root of the bzlmod example was breaking, when running under bazel coverage //:test. However, it started working when I renamed __init__.py to lib.py. I am suspecting that this has to do with the fact that the layer of indirection that coverage introduces could be something to do with that. Note that bazel test ... works regardless of file naming.

I think that all of these issues may be related to Python entrypoint template that is stored in this repository and not an issue with rules_python itself.

[adam-azarchs]

I'll admit to not having actually checked any test cases for a __init__.py module. TBH python's import processing still confuses me in some cases, especially around virtual packages. It's possible that the template is doing something wrong in that case. However, what did you have set for --incompatible_default_to_explicit_init_py? Because I could definitely see that messing things up.

And yes, you won't get coverage in subprocesses, because coverage works by installing its hooks into the interpreter process your code is running in, and it isn't going to install those in a subprocess. Not sure if there's a good way around that short of hacky attempts to detect when the process is running with coverage enabled, which seems like a bad idea.

[rafmagns-skepa-dreag]

@adam-azarchs there is a pytest plugin called pytest-cov that is able to get around this issue and will profile all subprocesses. this means that it will also work with pytest-xdist which is not compatible with bazel coverage right now. a call to coverage combine after coverage run and before coverage lcov I think should at least allow using this common plugin

[adam-azarchs]

Last I checked, though, pytest-cov couldn't output in LCOV format, which is a requirement.

[adam-azarchs]

(also pytest-cov requires pytest, which is a pretty heavy-weight dependency)

[rafmagns-skepa-dreag]

Ah sorry I was a bit confusing - I meant that it would allow using pytest-xdist which is quite common.

Pytest-cov is now able to output in lcov format just as coverage can. And you're absolutely right about pytest bring heavy.

What I really meant to suggest is that between pytest-cov and coverage-enable-subprocess there's probably enough info to make bazel compatible with these subprocesses during coverage automatically and natively if the desire is there. But adding a call to coverage combine would at least allow people to create their own workarounds to the problem. The combine command doesn't do anything if there are not multiple files to combine so would be safe for existing users

[JeroenSchmidt]

Hi All,
Has anyone in this thread come across any interesting workarounds and solutions to the original issue?
I can't decide if I should dive into the various workarounds suggested already or perhaps waiting a bit longer for a permanent fix/workaround to the issue.

[phst]

rules_python now supports coverage natively: https://rules-python.readthedocs.io/en/stable/coverage.html
Does that work for you?

[JeroenSchmidt]

Hi @phst
Coverage works however the coverage reports are empty so it's not possible to produce a merged coverage report across test targets.