Argument filtering for pytest fails to handle custom arguments

[omartin2010]

Issue Type: Bug

I created a fixture in conftest.py that reads at input args to pytest (using pytest_addoption. See code below. When I try and add this option to my settings.json file so that my tests execute properly, it fails with the ERROR: file not found...

Here is my conftest.py file :

import pytest
import os


def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")


@pytest.fixture(scope='session')
def config_file(request):
    config_file_value = request.config.option.config_file
    if config_file_value is None:
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    elif not os.path.exists(config_file_value):
        print(f'config_file_value = {config_file_value}')
        pytest.skip()
    return config_file_value

And then my settings.json relevant to this:

{
    "python.pythonPath": "/usr/bin/python3",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.cwd": "",
    "python.autoComplete.addBrackets": true,
    "python.testing.pytestArgs": [
        "--config_file app/config.json",
    ]
}

And finally the output of the console :

ython /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.6.9, pytest-5.4.2, py-1.5.2, pluggy-0.13.1
rootdir: /root/jetson-task-manager
plugins: forked-0.2, xdist-1.22.1
collected 0 items

============================ no tests ran in 0.06s =============================
ERROR: file not found: --config_file app/config.json


Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Running pytest version 5.4.2. Feels to me like this option is taken as if it were a file to test, versus an actual argument. Ideas? Am I missing something?

Extension version: 2020.4.76186
VS Code version: Code - Insiders 1.46.0-insider (d487078dc7fc1c276657cadb61b4f63833a8df55, 2020-05-07T16:19:54.327Z)
OS version: Windows_NT x64 10.0.18363
Remote OS version: Linux arm64 4.9.140-tegra
Remote OS version: Linux arm64 4.9.140-tegra

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-8650U CPU @ 1.90GHz (8 x 2112)
GPU Status	2d_canvas: enabled flash_3d: enabled flash_stage3d: enabled flash_stage3d_baseline: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: disabled_off protected_video_decode: enabled rasterization: enabled skia_renderer: disabled_off_ok video_decode: enabled viz_display_compositor: enabled_on viz_hit_test_surface_layer: disabled_off_ok webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.93GB (7.08GB free)
Process Argv
Screen Reader	no
VM	0%

Item	Value
Remote	SSH: jetson
OS	Linux arm64 4.9.140-tegra
CPUs	unknown (6 x 345)
Memory (System)	7.68GB (0.66GB free)
VM	0%

Item	Value
Remote	Container taskmanager:latest (/taskmana…
OS	Linux arm64 4.9.140-tegra
CPUs	unknown (6 x 1420)
Memory (System)	7.68GB (0.66GB free)
VM	0%

[karthiknadig]

@omartin2010 You have to split these into separate arguments. Right now it is treating --config_file app/config.json as a filename.

"python.testing.pytestArgs": [
        "--config_file", 
        "app/config.json",
    ]

[omartin2010]

if I do that, it doesn't work, I get this :

python /root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /root/jetson-task-manager -s --cache-clear --config_file app/config.json
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: unrecognized arguments: --config_file
  inifile: None
  rootdir: /root/jetson-task-manager


Traceback (most recent call last):
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/root/.vscode-server-insiders/extensions/ms-python.python-2020.4.76186/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

Turns out I also have another argument that needs to be there which is the directory that contains the tests, so my args are :

    "python.testing.pytestArgs": [
        // "test",
        "--config_file",
        "app/config.json"
    ]

With or without the test argument (not having this line at all or putting it last), I get the same error..

[karthiknadig]

Looking at pytest's help i don't see a config-file but i see a -c. I don't think they have a expanded config file option.

> py -3.8 -m pytest --help | findstr config
                        configuration file raise errors.
  -c file               load configuration from `file` instead of trying to
                        locate one of the implicit configuration files.
test session debugging and configuration:
  -h, --help            show help message and configuration info
  --trace-config        trace considerations of conftest.py files.

Try this:

"python.testing.pytestArgs": [
        "-c",
        "app/config.json"
    ]

[omartin2010]

I think I may have been unclear. This is for my own, user defined parameter, one one for pytest itself, so that a fixture can retrieve it in this kind of a function. Here is an extract of my conftest.py file:

def pytest_addoption(parser):
    parser.addoption("--config_file", action="store")

so that it's passed to my pytest session. So I want this param to pass straight to my stuff. So far it works when I run this from the command line, just not when from vscode...

[karthiknadig]

Looks like the bug is in filterArguments for pytest:

vscode-python/src/client/testing/pytest/services/discoveryService.ts

Line 46 in 61b179b

const args = this.argsService.filterArguments(options.args, TestFilter.discovery);

We pass the config_file arguments there and we don't get back the expected arguments. In the case of :

        "python.testing.pytestArgs": [
            "--config_file",
            "app/config.json",
            "tests"
        ]

Expected:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json tests

Actual:

python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file app/config.json
python c:\GIT\issues\s p\vscode-python\pythonFiles\testing_tools\run_adapter.py discover pytest -- --rootdir c:\GIT\issues\issue11736 -s --cache-clear --config_file tests

Not sure why it prints out the command twice with different arguments.

[Querela]

I have a similar issue, I also try to provide custom defined options to my tests and the discovery doesn't forward them correctly, so pytest raises an error and no tests will be found.

My setup is almost identical to the description above. In addition I allowed the config names to be specified via environment variables which is the only way to get it to work!

I had to default to "python.envFile": "${workspaceFolder}/.env", which worked in my case as I defaulted to environment variables if no explicitely provided. But not all tests will allow this...

# conftest.py
def pytest_addoption(parser):
    def environ_or_required(key: str) -> Dict[str, Any]:
        return (
            {"default": os.environ.get(key)}
            if key in os.environ
            else {"required": True}
        )

    parser.addoption(
        "--discord-token",
        type=str,
        help="discord bot token",
        **environ_or_required("DISCORD_TOKEN"),
    )
    parser.addoption(
        "--discord-channel",
        type=str,
        help="discord bot channel name or id",
        **environ_or_required("DISCORD_CHANNEL"),
    )

@pytest.fixture(autouse=True)
def env_setup(monkeypatch: pytest.MonkeyPatch, request: pytest.FixtureRequest):
    monkeypatch.setenv("DISCORD_TOKEN", request.config.getoption("--discord-token"))
    monkeypatch.setenv("DISCORD_CHANNEL", request.config.getoption("--discord-channel"))

// settings.json
{
    "python.pythonPath": "venv/bin/python",
    "restructuredtext.confPath": "",
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true,
    "python.testing.pytestArgs": [
        "--discord-token",
        "my-private-token",
        "--discord-channel",
        "Allgemein",
    ],
}

And one error message:

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: the following arguments are required: --discord-token, --discord-channel


Traceback (most recent call last):
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py", line 22, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/__main__.py", line 100, in main
    parents, result = run(toolargs, **subargs)
  File "/home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 44, in discover
    raise Exception("pytest discovery failed (exit code {})".format(ec))
Exception: pytest discovery failed (exit code 4)

I have to add that I have to restart VSCode else the changes in settings.json will not effect the current pytest discovery. With reloading, the pytest args are mangled ... I'm not sure how to provide them correctly ...

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token --discord-channel my-private-token
python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token --discord-channel Allgemein
Test Discovery failed: 
Error: ERROR: usage: run_adapter.py [options] [file_or_dir] [file_or_dir] [...]
run_adapter.py: error: argument --discord-token: expected one argument

Configuring it like so also fails: (sometimes with a different message...)

    "python.testing.pytestArgs": [
        "--discord-token 'my-private-token'",
        "--discord-channel Allgemein",
    ],

python /home/user/.vscode-server/extensions/ms-python.python-2020.12.424452561/pythonFiles/testing_tools/run_adapter.py discover pytest -- --rootdir /mnt/c/work/project/python-transformer-discord-notifier -s --cache-clear --discord-token 'my-private-token' --discord-channel Allgemein
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux -- Python 3.8.5, pytest-6.2.1, py-1.10.0, pluggy-0.13.1
rootdir: /mnt/c/work/project/python-transformer-discord-notifier, configfile: setup.cfg
plugins: cov-2.10.1
collected 0 items

========================= no tests collected in 0.07s ==========================
ERROR: file or directory not found: --discord-token 'my-private-token'

(Using terminal.integrated.env.linux for environment variables did not seem to work. Might be the wrong setting. Env file worked.)

Are others having the same issue?

[ralwing]

I'm also facing this issue.
Passing arguments:

  "python.testing.pytestArgs": [
    "-s",
    "--result-logger", 
    "local"],

Triggers error:

Error: Process exited with code 4: ERROR: usage: __main__.py [options] [file_or_dir] [file_or_dir] [...]
__main__.py: error: unrecognized arguments: --result-logger 
  test_simple_object_detection.py::test_simple_object_detection[roundabout_entrance]
  inifile: None
  rootdir: /home/gigi/workspace/test_harness/as_simulator_tests

[karthiknadig]

@ralwing that seems to be error coming from pytest itself. If you run that command manually in the terminal what happens?

[apassy]

Having a very similar problem:
"python.testing.pytestArgs": [
"-rP",
"--html=c:\Data\pytest-report.html",
"--self-contained-html"
],

These are all ignored - not matter what I do with them, they do not effect running tests either with "run test" or "debug test"

window 10, vs code 1.58.2
pytest, pytest-html (among others)

[wkerzendorf]

same issue

[wkerzendorf]

@karthiknadig any movement on this issue. It cropped up in regards with another issue as well

[karthiknadig]

We are in the process of re-writing the testing adapter #17242 . It is actively being worked on in the current milestone.

[wkerzendorf]

@karthiknadig - as a fellow open-sourcer - happy to play guinea pig 😉 - thanks for your work

[binishkaspar]

Until a fix available, you can add these options on pytest.ini file,

[pytest]
addopts = --config_file app/config.json

https://docs.pytest.org/en/6.2.x/customize.html

[marq4]

So basically I can't dynamically choose between two different implementations like:
python -m pytest --algorithm merge
python -m pytest --algorithm quick
To reuse the same tests?

[PaulWaldo]

Has there been any movement on this? I can't get any of the suggestions or workarounds to work (VS Code Version: 1.80.2)

[eleanorjboyd]

I created a proposal with a solution for this problem, would love your input on the above issue! #21845

[elongstreet88]

I was able to use this format to get around it:

{
    "python.testing.pytestArgs": [
        "--mycustom-arg1=value1",
        "--mycustom-arg2=value2"
     ]
}