Tests not discovered, but running tests anyway works fine.

[bigjools]

I've got a very simple unittest-based set up. My settings.json has this:

    "python.testing.unittestEnabled": true,
    "python.testing.unittestArgs": [
        "-s",
        "sofi.tests"
    ],

When doing test discovery I get this output:

Test Discovery failed: 
Error: spawn /path/to/my/python3 ENOENT

However I can run the tests just fine! I get the output in the Python Debug console.

/usr/bin/env /path/to/my/python3 /my/homedir/.vscode/extensions/ms-python.python-2021.6.944021595/pythonFiles/lib/python/debugpy/launcher 45627 -- /my/homedir/.vscode/extensions/ms-python.python-2021.6.944021595/pythonFiles/visualstudio_py_testlauncher.py --us=sofi.tests --up=test*.py --uvInt=1 --result-port=34281 
.........................................................s.....................
----------------------------------------------------------------------
Ran 79 tests in 0.311s

Can anyone shed any light on this?

(Running 1.57.1 on Ubuntu 20.04, Python extension version 2021.6.944021595)

[brettcannon]

If you're seeing ENOENT in the logs then that means VS Code can't access your Python interpreter for some reason which is not run in the terminal like the debugger is. Have you tried copying and pasting in the command that the extension runs for test discovery into your terminal to see the result?

[brettcannon]

I have no idea what testtools is. Do you have a link?

We specifically only support vanilla unittest and pytest, so the fact that you seem to be using a custom running may explain why we are not discovering what you want us to.

[bigjools]

Sorry, that was a thinko to use the testtools runner, however you can safely ignore that I am using testtools, its runner is a drop-in replacement for unittest, and everything runs fine with unittest:

.tox/py3/bin/python -m unittest discover
...............................................................s.....................
----------------------------------------------------------------------
Ran 85 tests in 0.291s

.tox/py3/bin/python -m unittest 
...............................................................s.....................
----------------------------------------------------------------------
Ran 85 tests in 0.199s

(FWIW: https://testtools.readthedocs.io/en/latest/
I find it far superior to pytest. In this case I am mainly using the rich suite of Matchers for test assertions. Testtools is used in large Python code bases like Launchpad and all over Openstack.)

Also please read here https://docs.python.org/3/library/unittest.html#load-tests-protocol
My loader is implementing a standard Python library protocol to load the tests. The load_tests function returns a Suite. I need to implement it to generate the scenario-based tests.

In other words, I think the code that vscode-python is using to enumerate tests is incorrect and likely needs to check whether it's been passed an iterable of Suites or not.

[brettcannon]

Oh, I'm fairly certain our test discovery code is incorrect. 😅 We are planning to rewrite it and will probably be asking you to test it out when it comes time to check that we got it right for your use case.

[bigjools]

I'll have a stab at seeing if I can make it work when I get some more spare time. I feel like there ought to be a more built-in way of determining what tests need running, after all the runner does it ok :)

[bigjools]

Also just as a point to note, the default TestLoader.discover in the Python library returns a single suite, so the fact that the plugin here is trying to iterate over multiple ones makes me wonder how it ever worked! If I remove my custom test loader, it still fails to pick anything up. Running the discover command myself in this situation also shows it returning a single suite.

[bigjools]

I've tested this alternative solution and it works well:

def generate_test_cases(suite):
    for test in suite:
        if isinstance(test, unittest.TestCase):
            yield test
        else:
            yield from generate_test_cases(test)

loader = unittest.TestLoader()
suite = loader.discover("<START DIR HERE>", pattern="test*.py")
print("\n".join(s.id() for s in generate_test_cases(suite)))

I'll submit a PR for this shortly, once I read the contribution guidelines etc.