Pytest running doesn't take into consideration env.

[CMLL]

Environment data

• VS Code version: 1.36.1
• Extension version (available under the Extensions sidebar): 2019.9.29489-dev
• OS and version: Ubuntu 19.04
• Python version (& distribution if applicable, e.g. Anaconda): 2.7.15+
• Type of virtual environment used (N/A | venv | virtualenv | conda | ...): virtualenv
• Relevant/affected Python packages and their versions: pytest
• Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info How to update the language server to the latest stable version #3977): LanguageServer

Expected behaviour

Test discovery and running have the enviroment variables declared in the .env file or the terminal.integrated.env.linux settings before executing.

Actual behaviour

Tests that depend on a specific environment variable being set to run fail.

Steps to reproduce:

1. Set a test that depends on a enviroment variable to run.
2. Declare that variable in an .env file or in terminal.integrated.env.linux
3. Run test.

Logs

Output for Python in the Output panel (View→Output, change the drop-down the upper-right of the Output panel to Python)

In this output I'm running a test that depends on a variable PANOPTA_INTEGRATION_TESTS_CONFIG to point to a file, and if not found it defaults to this file /etc/panopta_tests_support_data.yaml, which doesn't exist.
I have declared said variable in an .env file at my workspace root.
PANOPTA_INTEGRATION_TESTS_CONFIG=/home/cllamach/Panopta/classic/src/models/tests/integration/support_data_config.yaml

But when adding log statements to pytest run module I see it tries to load the default file instead of the one declared in the .env file. I ran into a similar problem with the PYTHONPATH variable not being taken into account when declared into the .env file and was able to bypass it by installing pytest-pythonpath to declare its values there.

python /home/cllamach/.vscode/extensions/ms-python.python-2019.9.29489-dev/pythonFiles/testing_tools/run_adapter.py discover pytest -- -s --cache-clear --disable-warning /home/cllamach/Panopta/classic/src/tests/
Test Discovery failed: 
Error: ============================= test session starts ==============================
platform linux2 -- Python 2.7.15+, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/cllamach/Panopta/classic/src/tests, inifile: pytest.ini
plugins: xdist-1.28.0, sugar-0.9.2, pythonpath-0.7.3, forked-1.0.2, pylama-7.7.1
ConfTest /home/cllamach/Panopta/classic/src/tests/integration/conftest.py
PKGPath /home/cllamach/Panopta/classic/src/tests/integration
Error import conftest [Errno 2] No such file or directory: '/etc/panopta_tests_support_data.yaml'
collected 0 items / 1 errors

==================================== ERRORS ====================================
________________________ ERROR collecting test session _________________________
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/py/_path/common.py:377: in visit
    for x in Visitor(fil, rec, ignore, bf, sort).gen(self):
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/py/_path/common.py:419: in gen
    if p.check(dir=1) and (rec is None or rec(p))])
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/_pytest/main.py:660: in _recurse
    ihook = self.gethookproxy(dirpath)
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/_pytest/main.py:481: in gethookproxy
    my_conftestmodules = pm._getconftestmodules(fspath)
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/_pytest/config/__init__.py:419: in _getconftestmodules
    mod = self._importconftest(conftestpath.realpath())
../../.virtualenvs/panopta_env/local/lib/python2.7/site-packages/_pytest/config/__init__.py:461: in _importconftest
    raise ConftestImportFailure(conftestpath, sys.exc_info())
E   ConftestImportFailure: (local('/home/cllamach/Panopta/classic/src/tests/integration/conftest.py'), (<type 'exceptions.IOError'>, IOError(2, 'No such file or directory'), <traceback object at 0x7f7a47389ef0>))
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
===================== 21 warnings, 1 error in 0.53 seconds =====================

Traceback (most recent call last):
  File "/home/cllamach/.vscode/extensions/ms-python.python-2019.9.29489-dev/pythonFiles/testing_tools/run_adapter.py", line 18, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/cllamach/.vscode/extensions/ms-python.python-2019.9.29489-dev/pythonFiles/testing_tools/adapter/__main__.py", line 90, in main
    parents, result = run(toolargs, **subargs)
  File "/home/cllamach/.vscode/extensions/ms-python.python-2019.9.29489-dev/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 36, in discover
    raise Exception('pytest discovery failed (exit code {})'.format(ec))
Exception: pytest discovery failed (exit code 2)

Output from Console under the Developer Tools panel (toggle Developer Tools on under Help; turn on source maps to make any tracebacks be useful by running Enable source map support for extension debugging)

XXX

[singleheart]

Same here:
VS Code version: 1.37.0-insider
Extension version (available under the Extensions sidebar): 2019.6.22090, 2019.6.24221, 2019.8.29288 (NOT until 2019.5.18875)
OS and version: CentOS 7.5.1804
Python version (& distribution if applicable, e.g. Anaconda): 3.6.x
Type of virtual environment used (N/A | venv | virtualenv | conda | ...): pyenv-virtualenv
Relevant/affected Python packages and their versions: pytest
Jedi or Language Server? (i.e. what is "python.jediEnabled" set to; more info #3977): LanguageServer

[DonJayamanne]

Unfortunately I'm unable to replicate this.
Here's what I'd like @singleheart @CMLL to try.

• Lets try a simple example to narrow things down.
• Create a new workspace folder
• Add a new file under the tests folder named test_one.py as follows:

import os

def test_case_one():
    with open("log.log", "w") as fp:
        fp.write(os.getenv("WOW", "not found"))

• Create a settings file .vscode/settings.json with the following contents:

{
    "python.testing.pytestArgs": ["tests"],
    "python.testing.unittestEnabled": false,
    "python.testing.nosetestsEnabled": false,
    "python.testing.pytestEnabled": true
}

• Create a .env file with the following contents WOW=1234
• Discover the tests and run the tests
• Please provide the contents of log.log
• If it doesnt work, please reload VS Code and let me know how that goes.

[DonJayamanne]

@singleheart What do you mean by the following.

Extension version (available under the Extensions sidebar): 2019.6.22090, 2019.6.24221, 2019.8.29288 (NOT until 2019.5.18875)

[singleheart]

How to replicate with Python extension version since 2019.6.22090:

• add a new folder "mypackage" in the workspace folder (in my case: /home/singleheart/work/test)
• add a new python file under the mypackage folder named "mymodule.py" as follows:

def foo():
     pass

• add a test function in test_one.py:

from mypackage.mymodule import foo
def test_case_two():
    foo()

• add new line in .env file (/home/singleheart/work/test is the location of my workspace)
  PYTHONPATH=/home/singleheart/work/test

Now test discovery failed with:

==================================== ERRORS ====================================
______________________ ERROR collecting tests/test_one.py ______________________
ImportError while importing test module '/home/singleheart/work/test/tests/test_one.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
tests/test_one.py:2: in <module>
    from mypackage.mymodule import foo
E   ModuleNotFoundError: No module named 'mypackage'
!!!!!!!!!!!!!!!!!!! Interrupted: 1 errors during collection !!!!!!!!!!!!!!!!!!!!
=========================== 1 error in 0.10 seconds ============================

Traceback (most recent call last):
  File "/home/singleheart/.vscode-server-insiders/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/run_adapter.py", line 18, in <module>
    main(tool, cmd, subargs, toolargs)
  File "/home/singleheart/.vscode-server-insiders/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/adapter/__main__.py", line 90, in main
    parents, result = run(toolargs, **subargs)
  File "/home/singleheart/.vscode-server-insiders/extensions/ms-python.python-2019.8.29288/pythonFiles/testing_tools/adapter/pytest/_discovery.py", line 36, in discover
    raise Exception('pytest discovery failed (exit code {})'.format(ec))
Exception: pytest discovery failed (exit code 2)

This problem did not happen before 2019.5.18875.

[DonJayamanne]

@singleheart please could you create a separate issue for what you are facing. It completely different.

[singleheart]

@singleheart please could you create a separate issue for what you are facing. It completely different.

I have created a new issue: #6915
Thank you.

<- This problem has been solved now. PYTHONPATH must be relative not absolute.

[CMLL]

@DonJayamanne Here is the results of testing this, which sadly result in the same behaviour.

In the following screen you can see the new workspace I created with the file structure on the explorer, the test case, workspace settings, .env file and resulting log file. Also set a breakpoint in the test and checked the contents of os.environ directly doesn't see WOW as a key.

'WOW' in os.environ.keys()
False

Edit:
Noticed I forgot to the python.envFile variable to the workspace settings file, so I did a second test with that in my settings, deleted the log file but got the same result.

[DonJayamanne]

@CMLL
Thanks for providing the information.

Please could you test this once again with the current development build.
See here for installation instructions https://github.com/microsoft/vscode-python/blob/master/CONTRIBUTING.md#development-build.
I believe this issue may have been resolved as part of another fix (relate to debugger).

When you run the tests how are you doing this? Are you debugging or just running them (what button/command are you using)? Or are you trying both options.

I ask as we fixed a bug related to env variables in the debugger recently.

[CMLL]

@DonJayamanne I installed version 2019.10.37374-dev from the link provided but still the same test case fails. I'm using the test test panel to discover and execute the tests. Using either the upper run all tests button and the Run Test link above the specific test fails. I tried with the Debug Test link and it also fails, but I normally don't use this to run tests.

[DonJayamanne]

Repro steps can be found here #7650

[nickcointracker]

Does test discovery take into account variables in .env? Or is it possible to specify an env file for test discovery? I see a problem where my module files (__init__.py) use env vars, and test discovery fails because during the discovery those variables are not defined.

[luabud]

@nickcointracker yes, it does when you specify the path to the .env file in your settings.json file. For example, if the .env file is in your workspace folder, you can add the following line there:

"python.envFile": "${workspaceFolder}/.env"

Does that help you?

[nickcointracker]

@nickcointracker yes, it does when you specify the path to the .env file in your settings.json file. For example, if the .env file is in your workspace folder, you can add the following line there:

"python.envFile": "${workspaceFolder}/.env"

Does that help you?

Thanks @luabud
I've tried setting that option, unfortunately it seems to be ignored during test discovery (although it works for regular python app execution). If it's supposed to be taken into account and vars be visible during the discovery phase, then I'll try to file a report with repro details.

[luabud]

@nickcointracker I believe it should, yes. Please do open an issue following the template there and providing repro steps. If this is the current expected behavior, then the team can just change the label to a feature request :)

[brgirgis]

I'm facing the same issue here. I change both PATH and PYTHONPATH in my env file on Windows to add few runtime dependencies. Somehow the PYTHONPATH change in the env file is making it through the debugger startup but not PATH. My suspicion is that the PATH forwarding is lost somehow in the child process layers to start up the debugger.

[luabud]

@brgirgis is the issue you're facing related to debugging tests or "regular" debugging?

[brgirgis]

From debug tests. Either from the annotations above the tests or from the dedicated GUI button to debug tests

[nealkruis]

I use a combination of conda and poetry to create virtual environments. Pytest is always listed as a development dependency in my project.toml file. After I run poetry install, pytest will be installed to the appropriate environment. When I switch between interpreters in my various environments, I would expect python.testing.pytestPath to default to the pytest in the same directory as the interpreter, but in fact it is using the pytest installed on my system.

I noticed that when I change my interpreter, the python.pythonPath variable does not change in my workspace settings file .vscode/settings.json. My understanding of the VS Code documentation on Python Environments, is that this should happen automatically when I select an interpreter.

I wonder if others facing this problem are seeing the same thing.

[karthiknadig]

@nealkruis check the logs to see if you are in the pythoPath deprecation experiment. If yes, then that might be the reason your settings json does not get updated.

Is your python.testing.pytestPath set to pytest or actual path? It should be set to just pytest, otherwise it will use whatever is set. We don't change this setting automatically.

[nealkruis]

@karthiknadig how will this appear in the logs? Is it in the Extension Host Log?

[karthiknadig]

You should see something like this, in the Output > Python:

[nealkruis]

Thanks! Yes, I "belong to experiment group 'DeprecatePythonPath - experiment'".

That said, what behavior should I expect related to pytest when I change interpreters? I would hope that VS Code would be smart enough to use the pytest from the corresponding interpreter's environment. If pythonPath is being deprecated, where do I find how the environment is being set within VS Code?

I can manually confirm that setting the pytestPath to be the same directory as the python interpreter resolves my issue. (Again, not clear it is the same issue described by the OP.)

[karthiknadig]

@nealkruis When in that experiment group, the selected python path is saved by the extension in an internal cache.

Python extension should use the pytest that is installed in the selected environment. The only reasons I can think of why this does not work is: 1) an explicit path is set for the 'pytestPath' setting 2) For some reason the environment is not activated.

One way you can work around this is start VS Code from an activated environment. That should allow the extension to pick the correct pytest, assuming no explicit path has been set using pytestPath setting. Also, your issue is unrelated to this one.

[SandoDev]

@CMLL
the way i solved it was to install pytest-dotenv

I ran the command
pip install pytest-dotenv

And ready!
Maybe you need to reload the vscode application

[mcauto]

I use pytest-env and tox.ini

[pytest]
env =
    key=value

[jmeachum]

@CMLL the way i solved it was to install pytest-dotenv

I ran the command pip install pytest-dotenv

And ready! Maybe you need to reload the vscode application

Thanks for this. It is a better solution for me, with the benefits that it ports over nicely when used for dev and CI/CD processes.

[chrisdel101]

What if it's a typed-in variable at run time that doesn't come from a file? How do I get the test runner to inject this? i.e. keyerror: ENV
The ENV from ENV=testing python3 -m unittest tests/tests.py

[niderhoff]

Hello, I am facing the same issue. On one computer it works, on another it doesn't. Both run VS Code insiders, updated daily. Both have exact same .vscode/settings.json.

one loads the .env variables correctly in integrated python (python test runnter), the other doesn't.

Any hints for debugging this?

EDIT: also installing python-dotenv does not alleviate this. I can confirm that python-dotenv works (when running pytest from commandline), but when I run the tests using the sidebar, it seems like VS Code is overwriting all env variables with some internal state, hence also resetting PYTHONPATH to "."

[JamesHutchison]

I'd say this has been a substantial time sink in my time using VS Code and configuring dev containers. I need a config specifically for the container, and this would be a simple solution if it actually worked correctly!

It's dead trivial to reproduce. Stick a library root anywhere non-standard and then try to use it with PYTHONPATH via the .env file.

I would say the terminal activating your virtual environment but not respecting .env is also a bit of an annoyance. If the whole thing isn't consistent then it begs the question what the point is.

[alsuarez]

I am facing a similar issue to #6891 (comment) where the PYTHONPATH variable in my .env file is not being read. For me, it works up to version 2023.8.0 of the extension. Starting from version 2023.10.0, it stops working.

I am able to replicate the issue like this:

1. Install any version of the extension starting from version 2023.10.0.

2. Create the python project with the following structure:

   .
   ├── .env
   ├── .vscode
   │   └── settings.json
   ├── lib
   │   ├── __init__.py
   │   └── b.py
   ├── pyproject.toml
   └── src
       ├── pkg
       │   ├── __init__.py
       │   └── a.py
       └── tests
           ├── __init__.py
           ├── conftest.py
           └── test_a.py
   

3. Add the following to your settings.json file:

   {
       "python.defaultInterpreterPath": "${workspaceFolder}/.venv/bin/python",
       "python.envFile": "${workspaceFolder}/.env",
       "python.linting.enabled": false,
       "python.testing.pytestEnabled": true,
       "python.testing.unittestEnabled": false,
       "python.testing.cwd": "${workspaceFolder}/src",
       "python.testing.pytestArgs": [
           "tests"
       ],
       "files.trimTrailingWhitespace": true,
       "files.trimFinalNewlines": true,
       "files.insertFinalNewline": true
   }
   

4. Add the following to the .env file:

   PYTHONPATH=./:./..
   

5. Attempt to import b from any of the test modules and configure testing in vs code to see it fail.

   1. If imported from the conftest.py file, you'll get a discovery error saying b can't be found.
   2. If imported from the test_a.py file, you'll get a test error saying b can't be found.

[niderhoff]

As far as I am aware, PYTHONPATH has never supported relative paths...?

[JamesHutchison]

It does. I have a bunch of repos with PYTHONPATH=. in the environment file

[eleanorjboyd]

Hello! Can you try this on the rewrite for testing and if it is still broken please attach logs from the Python output channel.

To use the rewrite yourself just add ”python.experiments.optInto": ["pythonTestAdapter"] to your user settings. You can confirm you have the rewrite enabled by setting your log level to trace, via the Developer: Set Log Level command in the command palette. Then check to see if Experiment 'pythonTestAdapter' is active is in your python logs.

Thanks!

[alsuarez]

@eleanorjboyd I tried this from the project described in #6891 (comment), but the issue persists.

First the following log messages are printed:

2023-12-12 20:31:57.077 [info] Discover tests for workspace name: python-vscode-bug - uri: /path/to/project
2023-12-12 20:31:57.077 [info] Running discovery for pytest using the new test adapter.

Then the following log messages are printed repeatedly:

2023-12-12 20:32:38.173 [debug] Clearing context for python dependencies not installed: ms-python.python.Python
2023-12-12 20:32:38.174 [debug] Found cached env for /path/to/project/.venv

[eleanorjboyd]

looks like it might be related to this: #22618

[alsuarez]

@eleanorjboyd I installed the pre-release version and test discovery is working for me now. Thanks! Which release is expected to include this fix?

[eleanorjboyd]

@alsuarez there is a point release out on stable now with the fix, it is version 2023.22.1. Glad that fixed it!