Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documentation should explain how to run tests #3260

Closed
yurivict opened this issue Oct 27, 2021 · 8 comments · Fixed by #3307
Closed

Documentation should explain how to run tests #3260

yurivict opened this issue Oct 27, 2021 · 8 comments · Fixed by #3307
Assignees
Labels
documentation Current issue related to documentation
Milestone

Comments

@yurivict
Copy link

yurivict commented Oct 27, 2021

The Testing Gensim section of the documentation doesn't explain how to run tests.

Running python -m pytest fails:

============================= test session starts ==============================
platform freebsd13 -- Python 3.8.12, pytest-4.6.11, py-1.9.0, pluggy-0.13.1
rootdir: /disk-samsung/freebsd-ports/textproc/py-gensim/work-py38/gensim-4.1.2
plugins: forked-1.0.2, cov-2.9.0, rerunfailures-10.1, timeout-1.4.2, xdist-1.32.0, mock-1.10.4, hypothesis-6.23.1, asyncio-0.10.0, localserver-0.5.0
collected 0 items / 44 errors

==================================== ERRORS ====================================
_______________ ERROR collecting gensim/test/test_aggregation.py _______________
gensim/matutils.py:1346: in <module>
    from gensim.corpora._mmreader import MmReader  # noqa: F401
E   ModuleNotFoundError: No module named 'gensim.corpora._mmreader'

During handling of the above exception, another exception occurred:
/usr/local/lib/python3.8/site-packages/py/_path/local.py:704: in pyimport
    __import__(modname)
gensim/__init__.py:11: in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
gensim/corpora/__init__.py:6: in <module>
    from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
gensim/corpora/indexedcorpus.py:14: in <module>
    from gensim import interfaces, utils
gensim/interfaces.py:19: in <module>
    from gensim import utils, matutils
gensim/matutils.py:1348: in <module>
    raise utils.NO_CYTHON
E   RuntimeError: Compiled extensions are unavailable. If you've installed from a package, ask the package maintainer to include compiled extensions. If you're building Gensim from source yourself, install Cython and a C compiler, and then run `python setup.py build_ext --inplace` to retry.
___________________ ERROR collecting gensim/test/test_api.py ___________________
gensim/matutils.py:1346: in <module>
    from gensim.corpora._mmreader import MmReader  # noqa: F401
E   ModuleNotFoundError: No module named 'gensim.corpora._mmreader'

During handling of the above exception, another exception occurred:
/usr/local/lib/python3.8/site-packages/py/_path/local.py:704: in pyimport
    __import__(modname)
gensim/__init__.py:11: in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
gensim/corpora/__init__.py:6: in <module>
    from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
gensim/corpora/indexedcorpus.py:14: in <module>
    from gensim import interfaces, utils
gensim/interfaces.py:19: in <module>
    from gensim import utils, matutils
gensim/matutils.py:1348: in <module>
    raise utils.NO_CYTHON
@mpenkov mpenkov added the documentation Current issue related to documentation label Dec 4, 2021
@mpenkov
Copy link
Collaborator

mpenkov commented Dec 4, 2021

Are you able to make a PR?

@piskvorky
Copy link
Owner

piskvorky commented Apr 19, 2022

@mpenkov how are the (local) tests run?

Another user bitten by this https://groups.google.com/g/gensim/c/MIbugjh2suc, CC @finnhacks42.

Let me know and I'll add it to the documentation. Both to Testing Gensim (where @yurivict looked for it) and also into our Developer page wiki.

@mpenkov
Copy link
Collaborator

mpenkov commented Apr 19, 2022

pytest gensim

@mpenkov
Copy link
Collaborator

mpenkov commented Apr 19, 2022

It also looks like the OP didn't compile the C extensions, which is why the test failed. So the problem wasn't running the tests, it was installing gensim from source.

@piskvorky
Copy link
Owner

piskvorky commented Apr 19, 2022

So, the recommended steps for a local test are now:

pip install -e .
pytest gensim

Correct?

@piskvorky
Copy link
Owner

piskvorky commented Apr 19, 2022

I tried that and got partial success.

Screen Shot 2022-04-19 at 17 27 41

  • Many tests were skipped. I looked around and the correct install command for local testing seems to be pip install -e .[docs,test]. (which fails on my machine = the latest MacOS on M1 silicon, with some install errors from nmslib, one of Gensim's dependencies)
  • Three tests failed outright.

I've updated the Developer page wiki, but the testing path is still not super clear to me. Might be due to my Apple M1 laptop.

@piskvorky
Copy link
Owner

piskvorky commented Apr 19, 2022

After some extra playing:

  • MacOS tests still broken. I tinkered with the CLANG flags and managed to install nmslib, and then pip install -e .[docs,test] worked. But after that, building the documentation with make html still fails:
    Screen Shot 2022-04-19 at 18 17 32
  • So I switched to Linux. It complained about a missing latex command, and after I installed that, a missing dvipng. After apting both – success!

@barracuda156
Copy link

barracuda156 commented Mar 19, 2024

Apparently tests are still broken on macOS?

--->  Testing py312-gensim
Executing:  cd "/opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_python_py-gensim/py312-gensim/work/gensim-4.3.2" && py.test-3.12 -o addopts='' 
============================= test session starts ==============================
platform darwin -- Python 3.12.2, pytest-7.4.3, pluggy-1.3.0
rootdir: /opt/local/var/macports/build/_opt_svacchanda_SonomaPorts_python_py-gensim/py312-gensim/work/gensim-4.3.2
plugins: cov-4.1.0
collected 0 items / 44 errors

==================================== ERRORS ====================================
_______________ ERROR collecting gensim/test/test_aggregation.py _______________
gensim/matutils.py:1352: in <module>
    from gensim.corpora._mmreader import MmReader  # noqa: F401
E   ModuleNotFoundError: No module named 'gensim.corpora._mmreader'

During handling of the above exception, another exception occurred:
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1310: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
<frozen importlib._bootstrap>:1360: in _find_and_load
    ???
<frozen importlib._bootstrap>:1331: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:935: in _load_unlocked
    ???
<frozen importlib._bootstrap_external>:995: in exec_module
    ???
<frozen importlib._bootstrap>:488: in _call_with_frames_removed
    ???
gensim/__init__.py:11: in <module>
    from gensim import parsing, corpora, matutils, interfaces, models, similarities, utils  # noqa:F401
gensim/corpora/__init__.py:6: in <module>
    from .indexedcorpus import IndexedCorpus  # noqa:F401 must appear before the other classes
gensim/corpora/indexedcorpus.py:14: in <module>
    from gensim import interfaces, utils
gensim/interfaces.py:19: in <module>
    from gensim import utils, matutils
gensim/matutils.py:1354: in <module>
    raise utils.NO_CYTHON
E   RuntimeError: Compiled extensions are unavailable. If you've installed from a package, ask the package maintainer to include compiled extensions. If you're building Gensim from source yourself, install Cython and a C compiler, and then run `python setup.py build_ext --inplace` to retry.
___________________ ERROR collecting gensim/test/test_api.py ___________________
gensim/matutils.py:1352: in <module>
    from gensim.corpora._mmreader import MmReader  # noqa: F401
E   ModuleNotFoundError: No module named 'gensim.corpora._mmreader'

During handling of the above exception, another exception occurred:
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:531: in collect
    self._inject_setup_module_fixture()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:545: in _inject_setup_module_fixture
    self.obj, ("setUpModule", "setup_module")
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:310: in obj
    self._obj = obj = self._getobj()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:528: in _getobj
    return self._importtestmodule()
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/local/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1387: in _gcd_import
    ???
. . .

UPD. Wait, maybe it is due to missing py-nmslib, I need to write a port for it and try again.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Current issue related to documentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants