Test: Pytest Rewrite

[eleanorjboyd]

Refs: (overarching issue) #17242

• macOS @jrieken
• linux @rzhao271
• windows @Yoyokrazy
• anyOS @rebornix
  Complexity: 5

Author: @eleanorjboyd

Create Issue

---

Thank you for doing this testing! I apologize for how long it is and please let me know if you have any questions!

Prerequisites:

1. Use the following repo of tests or use your own repo of pytests https://github.com/eleanorjboyd/inc_dec_example_repo

Background Information:
This rewrite is for the underlying infrastructure of how python handles testing therefore all behavior visible to the user should be pretty much the same (minus maybe some error logs going in a better place or test names shortened). Therefore throughout this testing, you can reference the current testing in Python to compare if you think something doesn't look right (or you can always post here too!). Secondly, debugging is not supported yet so there is no need to test debugging on the rewrite which will also be reflected in the TPI steps.

Steps:

Part 0: Setup

1. launch vscode with the environment variable ENABLE_PYTHON_TESTING_REWRITE = '1'
2. Now open the inc_dec_example_repo testing repo in vscode
3. use the create environment command from the command palette, select venv and whatever python version you want >=3.7
4. in the terminal run python -m pip install pytest
5. from the command palette run python: configure tests select pytest and "." (root directory)

Part 1: General

1. Go to the testing explorer, and confirm all the tests are discovered and nested correctly (please see attached image for reference on how it should look or compare it to the stable version of the extension since it will look the same)
2. There are two types of test cases in pytest, regular test cases and parameterized test cases (examples of both are in the test repo). Please try the following for both
   • Add a test case, reload
   • Remove a test case, reload
   • Run via the gutter
   • Run via the test explorer (individually as as part of a folder)
   • Run select, non-adjacent tests (cmd+click to select individual tests)
3. Now go to the output tab and select python: output logs, run tests and confirm that the output appears with all the necessary information (how many tests were run, which succeeded, explanation of failure if that occurred)

Part 2: Error Handling

1. Make one of your test files have a syntax error then run test discovery
2. Make an error with parameterization (run test, see if it errors there)
3. Discover tests, then make a syntax error, then try and run the test (check error message appears)

Part 3: Adding Pytest Args

1. Add -x to your pytest args in your settings.json and run the test folder, check that this arg worked (it works if it stops after a single test failure)
2. Try your own arg from here: https://docs.pytest.org/en/6.2.x/usage.html

Part 4: Additional Commands

1. Try out different commands from the command palette related to testing to see if they work
2. Try out the different items on the top righthand side of the testing panel such as collapse test cases and clear all results

test explorer reference image:

[Yoyokrazy]

Ran into same issue as #21124, had to fully reload the window to get tests appearing.