Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tox tests + branch coverage + coverage combine = internal error #168

Open
obestwalter opened this issue Sep 5, 2017 · 8 comments
Open
Labels

Comments

@obestwalter
Copy link
Member

obestwalter commented Sep 5, 2017

Hi,

I added coverage to the tests of the tox project and stumbled over a problem when trying to record branch coverage. The problem is os independent and can be reproduced easily with the code of the tox project.

versions:

coverage==4.4.1
py==1.4.34
pytest==3.2.1
pytest-cov==2.5.1

reproduce:

$ git clone git@github.com:tox-dev/tox.git
$ cd tox
$ pip install -e .
$ pip install pytest pytest-cov 
$ pytest tests/test_z_cmdline.py::TestToxRun --cov-branch --cov=tox

result:

========================================================== test session starts ===========================================================
platform linux -- Python 3.6.2, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
tox comes from: '/home/oliver/work/tox/tox/tox/__init__.py'
rootdir: /home/oliver/work/tox/tox, inifile: tox.ini
plugins: cov-2.5.1
collected 3 items                                                                                                                                                             

tests/test_z_cmdline.py ...
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "[...]/_pytest/main.py", line 110, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "[...]/_pytest/main.py", line 146, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 745, in __call__
INTERNALERROR>     return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 339, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook, methods, kwargs)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 334, in <lambda>
INTERNALERROR>     _MultiCall(methods, kwargs, hook.spec_opts).execute()
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 613, in execute
INTERNALERROR>     return _wrapped_call(hook_impl.function(*args), self.execute)
INTERNALERROR>   File "[...]/_pytest/vendored_packages/pluggy.py", line 250, in _wrapped_call
INTERNALERROR>     wrap_controller.send(call_outcome)
INTERNALERROR>   File "[...]/pytest_cov/plugin.py", line 235, in pytest_runtestloop
INTERNALERROR>     self.cov_controller.finish()
INTERNALERROR>   File "[...]/pytest_cov/engine.py", line 154, in finish
INTERNALERROR>     self.cov.stop()
INTERNALERROR>   File "[...]/coverage/control.py", line 809, in combine
INTERNALERROR>     self.data, aliases=aliases, data_paths=data_paths, strict=strict,
INTERNALERROR>   File "[...]/coverage/data.py", line 735, in combine_parallel_data
INTERNALERROR>     data.update(new_data, aliases=aliases)
INTERNALERROR>   File "[...]/coverage/data.py", line 488, in update
INTERNALERROR>     raise CoverageException("Can't combine line data with arc data")
INTERNALERROR> coverage.misc.CoverageException: Can't combine line data with arc data

======================================================= 3 passed in 17.21 seconds ========================================================

There are a lot of timestamped .coverage.* files and all except for the first one contain "arc" data - the first one contains "lines" data - which causes the error.

I have not yet tried to reproduce this with vanilla coverage as the reason I use pytest-cov is that there are a lot of tests that run in a subprocess and pytest-cov takes care of that. It might be that the problem is actually in coverage.py itself but I thought I report this here first.

pytest tests/test_z_cmdline.py::TestToxRun --cov=tox works.

@ionelmc
Copy link
Member

ionelmc commented Sep 6, 2017

Ah yes, you've hit the default coverage config problem.

Coveragepy is using ".coveragerc" as a magic value that means: use .coveragerc if it exists, otherwise lookup a bunch of other files for configuration. So we can't just make it an absolute path (that failing test is changing CWD btw)

I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?

@ionelmc ionelmc added the bug label Sep 6, 2017
@obestwalter
Copy link
Member Author

Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:

 pytest --cov-config=tox.ini --cov=tox

So that works and is good enough for me at the moment.

@ionelmc
Copy link
Member

ionelmc commented Nov 1, 2017

I guess we could provide a workaround: we make it absolute only if it's not ".coveragerc"?

Actually I'm not really sure what I meant, I think I wrote that while drunk. Can we close this?

sbellem added a commit to sbellem/bigchaindb that referenced this issue Dec 19, 2017
sbellem added a commit to sbellem/bigchaindb that referenced this issue Dec 19, 2017
@jinh-dk
Copy link

jinh-dk commented Jan 17, 2018

I saw the same error message. Do you have any ideas about the cause?

@ionelmc
Copy link
Member

ionelmc commented Jan 17, 2018

Well the cause is mismatched coverage configuration when subprocesses are used (eg: one process has branch coverage enabled while the other doesn't). If you have a reproducer I'd gladly* look at it.

* if it's minimal :D

@marscher
Copy link

Hi,
we have this problem with molPX. Unfortunately it has a lot of dependencies and uses conda-build for building/testing. But we are running into exact the same problem here. We have a .coveragerc file with branch coverage enabled, as well as parallel=True, because we use sub-processes (eg. jupyter kernels) to execute tests. This is performed via the pytest plugin nbval. The measurements are done by pytest-cov.
We are coping the .coveragerc file to the testing directory, so it should be in place. We then also tried to pass it directly as cov-config value, but it seems to be ignored at least once.

If you're willing to take a look, the stuff is here:
https://github.com/markovmodel/molPX/tree/test_

@blueyed
Copy link
Contributor

blueyed commented Nov 15, 2018

You can use export COVERAGE_DEBUG=process,config,dataio (or similar, see coverage.py docs) to debug this (look at "config_files", cwds etc).

jayvdb added a commit to jayvdb/py2many that referenced this issue Apr 15, 2021
jayvdb added a commit to jayvdb/py2many that referenced this issue Apr 15, 2021
jayvdb added a commit to jayvdb/py2many that referenced this issue Apr 15, 2021
jayvdb added a commit to jayvdb/py2many that referenced this issue Apr 15, 2021
@kdeldycke
Copy link

Hi @ionelmc - thanks. I don't quite understand how that workaround you proposed would work, but a workaround that works already is passing the path to the pytest-cov like:

 pytest --cov-config=tox.ini --cov=tox

So that works and is good enough for me at the moment.

Do we still need the workaround of passing explicitly --cov-config=pyproject.toml now that pytest is defining the root dir relative to pyproject.toml as of 8.1.0? I am referring to: pytest-dev/pytest#11962

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants