pytest 'slow' not a registered marker with --strict flag on other projects

[kmodry]

Create a file test_pandas.py with import pandas

$ echo 'import pandas' > test_pandas.py

run pytest with --strict flag

$ pytest --strict test_pandas.py
================================================ test session starts ================================================
platform darwin -- Python 3.6.1, pytest-3.1.0, py-1.4.33, pluggy-0.4.0
rootdir: /private/tmp, inifile:
plugins: xdist-1.15.0, repeat-0.4.1, cov-2.4.0
collected 0 items / 1 errors

====================================================== ERRORS =======================================================
__________________________________________ ERROR collecting test_pandas.py __________________________________________
test_pandas.py:1: in <module>
    import pandas
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/__init__.py:60: in <module>
    import pandas.testing
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/testing.py:7: in <module>
    from pandas.util.testing import (
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/pandas/pandas/util/testing.py:55: in <module>
    slow = pytest.mark.slow
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
/Users/taugspurger/Envs/pandas-dev/lib/python3.6/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
============================================== 1 error in 1.10 seconds ==============================================

(original post below)

---

Importing panda make my pytest tests fail because of one marker "slow" is required in
/usr/local/lib/python2.7/dist-packages/pandas/util/testing.py:55: in
slow = pytest.mark.slow

I don't use slow marker and have to add it to run my test even though it's just for panda

complete stack trace :

tests/_lib/createfullteststoresem.py:54: in createFullTestStoreSem
from oxlib.sem.history import get_ad_product_history
oxlib/sem/history.py:5: in
import pandas as pd
/usr/local/lib/python2.7/dist-packages/pandas/init.py:60: in
import pandas.testing
/usr/local/lib/python2.7/dist-packages/pandas/testing.py:7: in
from pandas.util.testing import (
/usr/local/lib/python2.7/dist-packages/pandas/util/testing.py:55: in
slow = pytest.mark.slow
/usr/local/lib/python2.7/dist-packages/_pytest/mark.py:184: in getattr
self._check(name)

---

[jreback]

you would have to show more information.

[TomAugspurger]

@kmodry can you provide more information? What version of pandas is installed? Is your project publicly available?

[koffie]

Hi, I just ran into the exact same problem while making a pull request on github for pyjson_tricks . This pullrequest caused travis to run the continues integration test, whose results you can see at: https://travis-ci.org/mverleg/pyjson_tricks/jobs/246885590 . According to that webpage the error occured with pandas version 0.20.2. An older test of the same project with pandas 0.19.2 at https://travis-ci.org/mverleg/pyjson_tricks/jobs/222351028 did not produce these problems.

[koffie]

To recreate the problem: clone https://github.com/koffie/pyjson_tricks (commit 4f5e734b70ffcb...) and run the tests as described in: https://github.com/koffie/pyjson_tricks/blob/master/docs/index.rst . I.e. just run either tox or detox from the root directory. Changing tox.ini to require pandas==0.19.2 doesn't produce the error, while requiring pandas==0.20.0rc1 or any higher version triggers the error.

[jreback]

@koffie still not entirely sure what this has to do with pandas. pls show a reproducible example with pandas code itself.

I would guess you are picking up the pandas conftest

[TomAugspurger]

Hmm, looking through @koffie's example, pandas fails to import with

(Pdb) import pandas.testing
*** AttributeError: module 'pandas' has no attribute 'plotting'

I'm not sure how that would cause the test error. Even install matplotlib in that env causes the same error.

[koffie]

Yes we are indeed picking up parts of pandas conftest and this is happening because there is the statement: import pandas.testing in the /pandas/__init__.py file. This means that you cannot import pandas without also implicitly importing the test code for pandas.

Now the error occurs because we like to test our code with the --strict option of pytest. And according to the documentation

Note
It is recommended to explicitly register markers so that:

• there is one place in your test suite defining your markers
• asking for existing markers via pytest --markers gives good output
• typos in function markers are treated as an error if you use the --strict option. Future versions of pytest are probably going to start treating non-registered markers as errors at some point.

Now we don't want to be forced to register all markers that are used in pandas.testing . So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas? This would solve all our problems and make it much easier to run tests in other projects that depend on pandas, but only want to run their own tests, and not of pandas.

[koffie]

Another solution if you really want to keep exposing all your testing functions is to write a pandas plugin for pytest as explained on https://docs.pytest.org/en/latest/writing_plugins.html#writing-plugins and register all your custom markers as explained on https://docs.pytest.org/en/latest/example/markers.html#adding-a-custom-marker-from-a-plugin .

[koffie]

By the way, a minimal way to reproduce the problem is the following:
create a file test_pandas.py containing only the line of code import pandas and then running pytest --strict test_pandas.py. It would be nice to be able to import pandas from a file on which you want to run pytest --strict.

[TomAugspurger]

@koffie thanks for the minimal example. Added to the original post.

So would it be possible to remove the importing of the pandas.testing code when one just wants to import pandas?

Don't think so. That's part of the public API, and we provide pandas.test() to run the unit tests.

I think you're suggesting to write a plugin is the way to go.

[koffie]

@TomAugspurger One could move the import pandas.testing statement inside the pandas.test() function. But I agree that a plugin is the better way to go.

[koffie]

By the way just running pytest --strict pandas also gives the same error. So maybe just registering the mark slow in setup.cfg would solve the problem. I don't know how fancy the marker registration framework of pytest is.

pytest --strict pandas
=========================================================================================== test session starts ============================================================================================
platform darwin -- Python 2.7.10, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /Users/mderickx/code/pandas, inifile: setup.cfg
collected 11472 items / 2 errors / 4 skipped 

================================================================================================== ERRORS ==================================================================================================
______________________________________________________________________________ ERROR collecting pandas/tests/test_sorting.py _______________________________________________________________________________
pandas/tests/test_sorting.py:19: in <module>
    class TestSorting(object):
pandas/tests/test_sorting.py:21: in TestSorting
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
________________________________________________________________________ ERROR collecting pandas/tests/groupby/test_value_counts.py ________________________________________________________________________
pandas/tests/groupby/test_value_counts.py:10: in <module>
    @pytest.mark.slow
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:251: in __getattr__
    self._check(name)
../../Library/Python/2.7/lib/python/site-packages/_pytest/mark.py:266: in _check
    raise AttributeError("%r not a registered marker" % (name,))
E   AttributeError: 'slow' not a registered marker
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 2 errors during collection !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
==================================================================================== 4 skipped, 2 error in 5.83 seconds ====================================================================================
bt-nac-h149:pandas mderickx$

[TomAugspurger]

Ha, I think that does it, at least for the simple example. Here's the diff if anyone wants to submit a PR before I have a chance tomorrow

diff --git a/pandas/util/testing.py b/pandas/util/testing.py
index 17e09b38b..d6ba95613 100644
--- a/pandas/util/testing.py
+++ b/pandas/util/testing.py
@@ -50,13 +50,6 @@ from pandas import (bdate_range, CategoricalIndex, Categorical, IntervalIndex,
 
 from pandas._libs import testing as _testing
 from pandas.io.common import urlopen
-try:
-    import pytest
-    slow = pytest.mark.slow
-except ImportError:
-    # Should be ok to just ignore. If you actually need
-    # slow then you'll hit an import error long before getting here.
-    pass
 
 
 N = 30
diff --git a/setup.cfg b/setup.cfg
index 8b32f0f62..e0d5f274a 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -27,3 +27,4 @@ split_penalty_logical_operator = 30
 testpaths = pandas
 markers =
     single: mark a test as single cpu only
+    slow: mark a test as slow

Thanks for the help @koffie

[koffie]

Hi,

Thanks for the pull request. I just tested this with 16797 applied. And it did fix the problem with running pytest --strict pandas since it reads the setup.cfg file. But if one installs this version of pandas with pip, and then tries pytest --strict test_pandas.py one still gets the same error about the marker slow not being registered. Apparently one really needs to write a plugin if one wants the markers to be registered even when running tests in a directory not containing your setup.cfg file.