Edited Discovery Logic

[eleanorjboyd]

closes #20078 and closes #20085 (which is about the testing work to support this code)

This logic now successfully works to discover the pytest repo tests. This branch takes into account all the previous comments made to the python discovery logic in the previous PR (located on my personal fork). Therefore this is a second round of edits on this code. It now works for the pytest library (discovers all the tests in the pytest library).

[brettcannon]

FYI one of your files doesn't pass Black formatting checks.

[brettcannon]

FYI this didn't pass Python type checking.

[brettcannon]

There are some inconsistencies in various things (e.g. docstring formatting, application of typing, etc.). It's always good to read through your changes to make sure there's consistency. It helps with readability as people know what to expect, as well as productivity as people's brain can tune to what to expect, having to think less about how to process something.

[karthiknadig]

Running pytest session to run other pytest session instances on windows seems to have weird problems:
Running it using microsoft store python causes this error:

Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python

Running it using python.org python causes this error:

Traceback (most recent call last):

  File "<frozen runpy>", line 198, in _run_module_as_main

  File "<frozen runpy>", line 88, in _run_code

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pytest\__main__.py", line 5, in <module>

    raise SystemExit(pytest.console_main())

                     ^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 189, in console_main

    code = main()

           ^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 147, in main

    config = _prepareconfig(args, plugins)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 328, in _prepareconfig

    config = pluginmanager.hook.pytest_cmdline_parse(

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_hooks.py", line 265, in __call__

    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_manager.py", line 80, in _hookexec

    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_callers.py", line 55, in _multicall

    gen.send(outcome)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\helpconfig.py", line 103, in pytest_cmdline_parse

    config: Config = outcome.get_result()

                     ^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_result.py", line 60, in get_result

    raise ex[1].with_traceback(ex[2])

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_callers.py", line 39, in _multicall

    res = hook_impl.function(*args)

          ^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1067, in pytest_cmdline_parse

    self.parse(args)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1354, in parse

    self._preparse(args, addopts=addopts)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\config\__init__.py", line 1237, in _preparse

    self.pluginmanager.load_setuptools_entrypoints("pytest11")

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\pluggy\_manager.py", line 287, in load_setuptools_entrypoints

    plugin = ep.load()

             ^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\importlib\metadata\__init__.py", line 202, in load

    module = import_module(match.group('module'))

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\importlib\__init__.py", line 126, in import_module

    return _bootstrap._gcd_import(name[level:], package, level)

           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1128, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed

  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\__init__.py", line [101](https://github.com/microsoft/vscode-python/actions/runs/4695498246/jobs/8324731667#step:20:102), in <module>

    from ._core._fileio import AsyncFile, Path, open_file, wrap_file

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\_core\_fileio.py", line 26, in <module>

    from .. import to_thread

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\to_thread.py", line 5, in <module>

    from .abc import CapacityLimiter

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\abc\__init__.py", line 81, in <module>

    from ..from_thread import BlockingPortal

  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load

  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked

  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\_pytest\assertion\rewrite.py", line 172, in exec_module

    exec(co, module.__dict__)

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\site-packages\anyio\from_thread.py", line 2, in <module>

    from asyncio import iscoroutine

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\asyncio\__init__.py", line 42, in <module>

    from .windows_events import *

  File "C:\hostedtoolcache\windows\Python\3.11.3\x64\Lib\asyncio\windows_events.py", line 8, in <module>

    import _overlapped

OSError: [WinError 10106] The requested service provider could not be loaded or initialized

We may have to figure out a different way to run these tests. May be just run them in a way we run them separately and collect data. Before running pytest session and then validate using pytest.

[karthiknadig]

These two errors on windows were hardest to debug:

[WinError 10106] The requested service provider could not be loaded or initialized

This is only on windows store python

_Py_HashRandomization_Init: failed to get random numbers to initialize Python

Looks like anyio package gets used if it exists when running pytest. That package seems to fail if you run a pytest sub session under pytest session. Moved the tests around so we run all the python tool tests before we install other packages. This only affects cases where we are testing the test framework itself like in this case testing pytest plugin.

[karthiknadig]

Thanks to @int19h we finally figured out what was going here. For any one ese that runs into this issue, basically USERPROFILE and SYSTEMROOT are required environment variables for python in windows. If it is missing it can cause errors like above. The pytest session helper script used subprocess.run did not copy the environ variables from the parent process. It set a minimal one that was needed for test, which did not include any of the system variables. This was the cause of the issue.

inconsistencies have been address and verbal OK was given