TPI: debugging, multiroot, and large workspace testing with pytest

[eleanorjboyd]

Refs (overarching issue): https://github.com/microsoft/vscode-python/issues/19098 and

• macOS @meganrogge
• linux @karthiknadig
• windows @cwebster-99
• anyOS @rebornix
  Complexity: 2

Author: @eleanorjboyd

Create Issue

Create Issue

---

Prerequisites:

1. Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information:

This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!).

Steps:

Part 0: Setup

1. make sure you have the setting "python.experiments.optInto": ["All"], in your user settings json
2. set the log level to trace
3. go to the python output channel and search for pythonTestAdapter and something should come up that looks like this: 2023-05-26 11:12:04.297 [info] Experiment 'pythonTestAdapter' is active (this confirms you are actively in the experiment)
4. Now open the inc_dec_example_repo/inc_dec_example_test_suite  testing repo folder in vscode
5. use the create environment command from the command palette, select venv and whatever python version you want >=3.7
6. in the terminal run python -m pip install pytest
7. from the command palette run python: configure tests select unittest and "." (root directory) and test_*.py

Part 1: Debugging

1. Go to the testing explorer, and confirm all the tests are discovered and nested correctly by file structure.
2. add some break points to the pytest tests and check to make sure these break points are hit
   1. make sure you can do other normal debugging actions (stop, restart, step over, etc)

Part 2: Large repo

1. If you have a very large python repo definitely try it on that repo but if not you can use the provided repo inc_dec_example_repo/inc_dec_example_test_suite
   1. if you are using the provided repo: in the test_parameterized.py file, change @pytest.mark.parametrize("num", range(1, 100)) to be much larger like @pytest.mark.parametrize("num", range(1, 10000))and make sure it both works for discovery and run tests (tests should be displayed on the test explorer after discovery and run)
   2. if you are using your own: make sure all tests can be discovered and run as expected within your large repo when set to use pytest

[eleanorjboyd]

Just pushed a change to the example repo since I edited the parameterized.py file last week but forgot to update the remote. If you already have the repo downloaded and don't want to pull, then here is the code that needs to be in that file:

@pytest.mark.parametrize("num", range(1, 100))
def test_odd_even(num):
    assert True

[eleanorjboyd]

An issue already arose with debugging as @cwebster-99 mentioned here, please add comments to the issue if you see this behavior as well. Thanks!

#21325

[cwebster-99]

Everything except debugging worked as expected for me
#21325

[rebornix]

I can't get test configure to work it seems. On the first run I saw

and error message: pytest not found.

After window reload, I no longer see the error message but the test view is always blank (with welcome content) even if I configure tests again

[eleanorjboyd]

@rebornix can you create an issue and provide logs from the python output channel and see if there is anything in the python test logs output channel too

[rebornix]

@eleanorjboyd I created #21343

[jrieken]

go to the python output channel and search for pythonTestAdapter and something should come up that looks like this

Hm, not for me... Using latest insiders and python prerelease I am seeing this

[meganrogge]

I don't see pythonTestAdapter in the output, but I do see Test server started. I believe this is bc even though I tried to set the log level to trace, it seems it's stuck on the debug level