launch.json "args" don't get passed to pytest unittests

[shaananc]

Environment data

• VS Code version: 1.48.0-insider
• Extension version (available under the Extensions sidebar): pyvsc-run-isolated.py
• OS and version: macOS 0.15.6 Beta (19G71a)
• Python version (& distribution if applicable, e.g. Anaconda): 3.8.2
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): poetry
• Relevant/affected Python packages and their versions: pytest

Expected behaviour

A configuration like the following:

    {
      "name": "Debug Tests",
      "type": "python",
      "request": "test",
      "console": "integratedTerminal",
      "osx": { "args": ["--pdb -s"] }
    }

should pass the arguments to pytest or the other testing framework used

Actual behaviour

Arguments get passed to pyvsc-run-isolated.py

Steps to reproduce:

1. Create a pytest compatible unit test

2. Create launch.json containing

    {
      "name": "Debug Tests",
      "type": "python",
      "request": "test",
      "console": "integratedTerminal",
      "osx": { "args": ["--pdb -s"] }
    }

3. Run an individual test

Note: I took a look at the code that runs pyvsc-run-isolated.py and the launch.json args just aren't supported yet as far as I can tell. Would hopefully not be too much work for someone familiar with vscode internals.

[brettcannon]

Thank you for the suggestion! We have marked this issue as "needs decision" to make sure we have a conversation about your idea. We plan to leave this feature request open for at least a month to see how many 👍 votes the opening comment gets to help us make our decision.

[jepperaskdk]

Is it documented somewhere how to pass arguments when running tests? And can it be configured both for Running and Debugging tests?

[karthiknadig]

@jepperaskdk "python.testing.unittestArgs": [] should send the arguments to the unittest runner in both with and without debugging scenario. Is there something else you are looking for?
https://code.visualstudio.com/docs/python/testing#_unittest-configuration-settings

[jepperaskdk]

@jepperaskdk "python.testing.unittestArgs": [] should send the arguments to the unittest runner in both with and without debugging scenario. Is there something else you are looking for? https://code.visualstudio.com/docs/python/testing#_unittest-configuration-settings

@karthiknadig I was wondering if I could configure arguments for both running and debugging, individually. E.g. our integration tests can run non-headless with a browser which is useful when debugging a test that is failing.

Also, is this issue valid? Is launch.json considered at all, when running/debugging tests?

[karthiknadig]

@jepperaskdk when it comes to testing using the testing UI, the launch.json support there is minimal. It can be used to send some additional flags to the debugger itself. But the args or other details vary test-to-test and even test-case-to-test-case. Configuring those for different scenarios is not supported if you depend on the test UI.

But if you want to setup your project to run tests entirely via launch json, then you can have full control over every aspect of the test run. Just that you won't see any updates or changes in the test UI. If you are ok going down this route and looking at the results in the terminal, then you can configure tests like this:

     {
            "name": "Run Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "noDebug": true, // <-- this will use the debug session but run the test without debugger injected into the process
            "args": []
        },
        {
            "name": "Debug Pytest",
            "type": "python",
            "request": "launch",
            "module": "pytest",
            "args": []
        }

You can pass any args, create any number of these configurations for each testing scenario, change terminal type, pass different env variables for testing vs running, etc. Just that the output will show up in the terminal.

[roblourens]

Some thoughts-

I hadn't realized that configuring the test explorer via launch.json is something that you can already do, via the "purpose": [ "debug-test" ] flag. Do you read all the config's parameters to build the config that will be used? I get the struggle of having to duplicate all of this configuration, if this would live somewhere else. But the config with "purpose" is not a real launch.json config because it can't be run on its own by the vscode debugger. It bugs me that the docs suggest a launch config that isn't hidden and won't run on its own- https://code.visualstudio.com/docs/python/testing#_debug-tests. If I set up a config this way, half of those properties are essentially placeholders- name, request, program, and console don't do anything, right? I would find it confusing as a user what I am actually configuring and why I need this boilerplate. I think all of that shows that launch.json is there to configure a feature which is similar to what you're using it for, but is a different feature.

I would feel a little better about it if it was at least possible to create a config that could be used as a normal debug config and also could be plugged into the test explorer ui. Then a user would not have to duplicate their variables for running in two ways. But I can't get this to work, if I add "module": "pytest" to the config with "purpose", it complains. I think it would also be confusing that there can be multiple configurations tagged like this but no way to pick which one is actually used. And the fact that this only makes sense if the user sets the flag to hide this config from the UI is an indication that something is off here.

In general I'd say that extensions shouldn't be reading and writing launch.json (or settings.json or other files that are managed by vscode). We provide API for interacting with these in the ways that we expect extensions to. I see #18778 that debug-test configs are not picked up from the workspace file, you can also keep these in settings.json under the "launch" key. That's a lot of places for you to keep track of and vscode could always change how this works.

I'll keep thinking about it but I think it would be a better user experience for this configuration to live in settings.json where you can control every aspect of it and don't have to create a "dummy" launch config to hold data. Some inspiration is maybe the terminal profiles setting where you give a list of terminal profiles and define all of their attributes. You can have run/debug/discovery attributes right next to each other in a common format. The only difference is that debugging tests outside of the test explorer is still set up in launch.json. I am fine keeping the "purpose": [ "debug-test" ] mechanism since it exists already and I think it would be a nice power-user feature if I can make that config also run on its own.

[tboddyspargo]

I came across this issue because we have some custom pytest command line flags that change the behavior of tests. While, we'd like to move away from this, today my team would like a mechanism to provide this custom flag to the command that VSCode runs when we choose to debug a specific test. AFAIK, the only way to do this today, is to add the option to the global python.testing.pytestArgs array, but that has a much broader impact that seems necessary. If the launch.json configuration options for debug-test supported the use of args (either overriding or appending to the list from python.testing.pytestArgs), then we would have more targeted control over the behavior.

Another example use-case is that I might want to include the --disable-warnings option for normal test runs (using python.testing.pytestArgs), but when running tests using the debugger, it might be useful to see warnings and add an extra -vv for extra verbosity. It's worth noting that the former would require being able to override python.testing.pytestArgs using the launch configuration.

Thanks for thinking about this issue! I'm definitely hopeful that there's a highly flexible way to allow both intuitive and targeted configuration of these behaviors.

[eleanorjboyd]

Hi! We are redesigning how args work in relation to test run and debug. Follow along here: #20425

This should resolve the above problem and create a clear flow for args and debugging so please contribute your thoughts to the discussion if you have any. Thanks!