Inconsistent behaviour when running parameterized tests via code lens and menu bar

[major-mayer]

Type: Bug

Behaviour

When I start a pytest-test the uses a parameterized fixture from the code lens (or however the small icon next to the line number is called) test, it starts the test session twice.
This doesn't happen when you start the same test from the menu-bar.
For me this is critical because I create some docker containers on each start of the test session, which doesn't work any more on the second try, because of conflicting names.

I would like the menu-bar behaviour to be used for both cases.

At first i though this could be related to this issue: #6043
But as I am using pytest and not Nosetest, I don't think that this is the case, even though it sounds similiar.

Expected vs. Actual

Expected: Even though the fixture is parameterized, the test session should only be started once.

Actual: Two separate test sessions are started.

Steps to reproduce:

Create the following file

import pytest


@pytest.fixture(params=["FULL", "PREVIEW"])
def quality(request):
    yield request.param


def test_soos(quality):
    print(f"Executing test. Quality={quality}.")

2. Start using the code lens. The follwing is generated in the test log:

============================= test session starts ==============================
platform linux -- Python 3.9.15, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/path, configfile: pytest.ini
plugins: anyio-3.6.2, dotenv-0.5.2
collected 1 item

tests/job/test_soos.py .                                                 [100%]

------------- generated xml file: /tmp/tmp-1196757sayGXK05wpG.xml --------------
============================== 1 passed in 0.01s ===============================
============================= test session starts ==============================
platform linux -- Python 3.9.15, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/path/, configfile: pytest.ini
plugins: anyio-3.6.2, dotenv-0.5.2
collected 1 item

tests/job/test_soos.py .                                                 [100%]

------------- generated xml file: /tmp/tmp-119675Qd67Md7BrYkZ.xml --------------
============================== 1 passed in 0.01s ===============================

3. Start using the menu bar. The following is generated:

============================= test session starts ==============================
platform linux -- Python 3.9.15, pytest-7.2.2, pluggy-1.0.0
rootdir: /home/laurenz/ModuleWorks/checkouts/ngsp-web-api-worktrees/ngsp-web-api, configfile: pytest.ini
plugins: anyio-3.6.2, dotenv-0.5.2
collected 2 items

tests/job/test_soos.py ..                                                [100%]

------------- generated xml file: /tmp/tmp-119675VTkFwCou66nq.xml --------------
============================== 2 passed in 0.01s ===============================

Diagnostic data

• Python version (& distribution if applicable, e.g. Anaconda): 3.9.15
• Type of virtual environment used (e.g. conda, venv, virtualenv, etc.): Venv
• Value of the python.languageServer setting: Pylance

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

XXX

User Settings

languageServer: "Pylance"

linting
• flake8Enabled: true

formatting
• autopep8Args: "<placeholder>"

testing
• pytestArgs: "<placeholder>"
• pytestEnabled: true

Extension version: 2023.4.1
VS Code version: Code - OSS 1.76.2 (ee2b180d582a7f601fa6ecfdad8d9fd269ab1884, 2023-03-16T10:00:00.439Z)
OS version: Linux x64 6.1.21-1-MANJARO
Modes:
Sandboxed: No

System Info

Item	Value
CPUs	Intel(R) Core(TM) i5-8600K CPU @ 3.60GHz (6 x 4200)
GPU Status	2d_canvas: enabled canvas_oop_rasterization: disabled_off direct_rendering_display_compositor: disabled_off_ok gpu_compositing: enabled multiple_raster_threads: enabled_on opengl: enabled_on rasterization: enabled raw_draw: disabled_off_ok skia_renderer: enabled_on video_decode: disabled_software video_encode: disabled_software vulkan: disabled_off webgl: enabled webgl2: enabled webgpu: disabled_off
Load (avg)	2, 3, 3
Memory (System)	31.26GB (15.72GB free)
Process Argv
Screen Reader	no
VM	0%
DESKTOP_SESSION	plasmawayland
XDG_CURRENT_DESKTOP	KDE
XDG_SESSION_DESKTOP	KDE
XDG_SESSION_TYPE	wayland

[major-mayer]

The same is happening, when you select multiple test using CTRL from the menu bar and start them by pressing the play button:

In this case, VSCode also launches multiple test sessions.

[eleanorjboyd]

Hello! We are currently working on a rewrite of testing in our extension. You can follow along on this issue and the rewrite should solve your problem as well. Thanks!

[major-mayer]

Hi @eleanorjboyd
Thanks for the hint.
I hope that the issue will finally get some traction, because it seems to be quite stale since Sep. 2021.
But if it will solve my problems, this would be nice.
I will keep this issue open for reference.

[eleanorjboyd]

Hello! We have just finished our testing rewrite and are beginning the roll out to users. If you are able, it would be very helpful to know if your issue still exists on the rewrite! To try it yourself, you need to be on the most recent release of the Python extension and add this setting to your users settings.json "python.experiments.optInto": ["pythonTestAdapter"].

You can confirm you have the rewrite enabled by setting "python.analysis.logLevel": "Trace", in your user settings then check for Experiment 'pythonTestAdapter' is active in your python logs.

Let me know if the rewrite fixes your issue. Thanks!

[brettcannon]

Because we have not heard back with the information we requested, we are closing this issue for now. If you are able to provide the info later on, then we will be happy to re-open this issue to pick up where we left off.