Option to force Poetry to create a virtual environment, even if a virtual env is active

[chernals]

I am using Poetry from within a conda environment; with Poetry being installed by conda (poetry is present in the environment.yaml file). This might not be ideal but for a specific setup this seems to work well.

When running poetry install, a venv is not created because Poetry detect that a virtual environment (the Conda one) is already active.

I believe that we should be able to force Poetry to create a virtual environment anyway: the Conda environment is more "physical" than virtual in this case, as it replaces the system Python.

The same issue occurs when doing poetry shell from within the Conda environment, after creating the virtual environment with the following workaround: I deactivate the Conda environment, use the full path to Poetry (in the bin directory of the Conda environment) and do poetry install.

[chernals]

This is related to #1724 and to #4050, but in the form of an explicit request to add an option.

[mforbes]

What would really be nice is a way to have an active underlying conda environment upon which poetry could create a virtual environment for installing dependencies while maintaining access to the underlying conda environment.

A use-case is a group of students working on a shared computer (like an HPC cluster): I would like to create a fairly full featured shared conda environment (called work here) which students have read access to, but not write. Currently poetry will treat this active conda environment as a virtual environment, but fail because it has no write access.

The behaviour I would like (if possible) is for poetry to be able to install or upgrade any missing dependencies as needed, akin to how pip install --user would, but to install these into a custom virtual environment rather than the users ~/.local.

I know that I could do this by manipulating paths PYTHONUSERBASE etc. but it would be really nices if this could "just work" in a way that consistent with general poetry usage.

Thus, daily work looks like:

conda activate work  # Activate read-only conda environment
poetry shell         # Creates a new virtual environment with access to `work`
python ...           # Works for anything using just `work`
poetry add ...       # If something is not in `work` (currently fails because of permissions)
poetry install       # Installed into the virtual environment like pip install --user

This allows students to get to work as quickly as possible, allowing us to provide most
of what they need in the work environment, but providing them a way to install other
libraries if needed. Everything new needed is recorded in pyproject.toml, so in
combination with the environment file for the work environment, this satisfies the
basic requirements for reproducibility.

When preparing for release, one would add the minimal set of missing dependencies
supplied by work into a clean virtual environment:

poetry env use 3.8
poetry shell         # Clean environment without `work` 
poetry add ...       # Clean up all dependences for release.
...

Now students need to work harder to properly define a clean minimal set of dependencies,
but this task does not stand in the way of them getting started.

Any suggestions for alternatives would be appreciated.

Edit: It looks like the a project-specific virtualenvs.options.system-site-packages = true config option as requested in #2937 might solve my issue.

[mforbes]

One other feature that would be really nice is if poetry shell could activate whatever environment is needed, including calling conda activate. I am not sure how internally this works, but the following workflow would be very useful:

conda activate my_conda_env
poetry env use python        # Tell poetry to use the my_conda_env
poetry shell                 # This works.

After doing this, it would be very nice if this or something similar could store information about the choice of environment in poetry.toml or something so that running poetry shell from within the project first activates the my_conda_env (until poetry env use or similar is called in the future).

[reitzig]

I don't use conda, but I second this feature request anyways.

In trying to debug a failing CI pipeline, it helps a lot if the venv is exclusive to the current build. On traditional Jenkins agents, that can only be assured by creating a venv in the current workspace, no matter which environments exist.
(One might say it's essential that the venv is exclusive to each build, always.)

Currently, I have no way (?) to force poetry to not use an existing environment (which I can't delete), which may or may not be the root of my problem.

This is of particular concern since running, say, end-to-end tests with poetry run myscript (with myscript declared in pyproject.toml:tool.poetry.scripts) doesn't work after poetry install --no-root; that is, I will have to pollute my "global" virtual environment with the state of the code of the current build, which leads me to believe that concurrent builds of, say, different branches won't work reliably.

PS: Not sure whether poetry install --no-root && poetry run myscript should be a bug report or feature request.

[tmke8]

I am also missing this feature, as I am migrating some code from pipenv to poetry. pipenv has PIPENV_IGNORE_VIRTUALENVS which has exactly the effect that's wished for here.

[JayZar21]

I described a workaround in this answer:

https://stackoverflow.com/questions/70739858/how-to-create-a-brand-new-virtual-environment-or-duplicate-an-existing-one-in-po

However, it would be nice to have the possibility to define more pyproject.toml and create other environments with the framework.

[tony]

This makes sense. I've had this happen to me again today

pipenv(1) does this behavior

IMO, In this case this would still be valid poetry usage, even if poetry wasn't necessarily installing the packages.

[popadi]

Any more updates/traction on this?

My usecase is upgrading some projects from py3.7 to py3.10 and it's really annoying having two conda envs for each project. It will greatly help forcing poetry to create a local env every time.

[Aperyon]

Deleting the existing virtualenv directory did help me.

$ poetry run python -m site
sys.path = [
    ...
    '/Users/user-name/Library/Caches/pypoetry/virtualenvs/some-project/lib/python3.10/site-packages',
]

After this I just simply ran

poetry install

and a new virtualenv env was automatically created.

I also added these lines to the project's poetry.toml (even though these are my global configs as well)

[virtualenvs]
create = true
in-project = true

[jaklan]

+1 for the conda case, it's pretty problematic in our remote environment, where python interpreter is provided by conda and poetry install doesn't create project-specific venvs...

[Green-li]

+1 for the conda case, it's pretty problematic in our remote environment, where python interpreter is provided by conda and poetry install doesn't create project-specific venvs...

I hava found PDM, which meets my requirements.

[coccoinomane]

I have found PDM, which meets my requirements.

Wow, PDM is like npm for Python... thanks for sharing!

[renan-r-santos]

I faced this problem as well and created a Poetry plugin that fixes this issue for the Conda use case. Although not ideal, this solution seems to work well when I tested it.
Would Poetry maintainers be open to adding a similar config such as virtualenvs.ignore-conda-env to Poetry? I'd be happy to contribute with a PR.

[TylerGrantSmith]

Referencing an earlier comment from @Secrus: #7878 (comment).

...My guess would be that you have Poetry installed in the same environment as your project, which is discouraged and prone to breakage when your dependencies clash with Poetry's.

This current behavior seems very much in contradiction to the linked guidance. If installing poetry in the same environment is to be discouraged, then attempts to initialize/install a project co-located with poetry should at the very least raise a warning that this is problematic.

[ndevenish]

I faced this problem as well and created a Poetry plugin that fixes this issue for the Conda use case

poetry-conda has fixed the issue I had with this. Poetry was installed on a (non-virtual) python, but I had a conda environment enabled that didn't contain python for some binary tooling (specifically h5ls and othe inspection tools). Every time trying to e.g. poetry run morgul-prepare it would say

The virtual environment found in /path/to/activated/conda/ENV seems to be broken.
Recreating virtualenv morgul-zttW3Bd7-py3.11 in /poetry_cache_path/virtualenvs/morgul-zttW3Bd7-py3.11
Warning: 'morgul-prepare' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.

The support to run uninstalled scripts will be removed in a future release.

Run `poetry install` to resolve and get rid of this message.

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/dls/science/users/mep23677/modules/python/3.11.3/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/dls/science/users/mep23677/jungfrau/jungfrau-commissioning/morgul/morgul_prepare.py", line 7, in <module>
    import h5py
ModuleNotFoundError: No module named 'h5py'

Then run poetry install, which appears to succeed, then the poetry run .... goes back to the above error message.

Poetry definitely shouldn't be trying to use an existing conda environment that doesn't contain python, and I never want it to, so a systemwide setting that lets's me turn off poetry trying to do this would be perfect.

This is a national lab, so we don't generally have free reign to install system-level packages, and using up-to-date things are usually a mixture of environment modules or conda environments.

[gabyx]

The behavior that poetry always prefers an acitvated venv is harmful and leads to stupid workarounds and nonsensical behavior. Normally also normal users have virtual envs on their machine which act as a basis for lets say python 3.10 which is active in the shell.

We can only force creating a venv by doing:

(unset VIRTUAL_ENV &&  poetry -v install)

which will then read the poetry.toml which has virtualenvs.create=true which then installs into ./.venv and not use the already activated one.

The above workaround however leads to issues with other commands because its not synchronized:

poetry env info --path # might be used in CI scripts to activate the correct venv ... etc etc

will in the same shell not give you the just created , nameley ./.venv but the system venv.

I am not even get the option POETRY_VIRTUALENVS_CREATE in that sense. If you specify true it should always create one, full stop. Because the other behavior "it always prefers already activated venvs" leads to really peculiar side effects in CI and on your local machine.

Please fix this in making this easy and comprehensible.

[bhelabhav]

Encountered this issue as well - would be great to have either:

1. Some opt-in/opt-out flag
   OR
2. Prompt when running poetry init or poetry install. This way users aren't surprised. e.g. It seems like you have an existing virtualenv active (/path/to/venv). Would you like to use this virtualenv or create a new one?

Without it, you can accidentally install things in a virtual environment you didn't expect to.

[maflAT]

I just tried using poetry for the first time following the introduction guide in the documentation.
The official documentation is very misleading and contradicting itself, which has brought me here!

In the very first chapter you see this big, red warning:

If you heed this warning, installing poetry in its own environment, and continue with the example you will run into the issue described here in the very next chapter.
Poetry will happily use its own environment for the example project without any question or warning.

Using your virtual environment

By default, Poetry creates a virtual environment in {cache-dir}/virtualenvs.

This is just plain wrong at this point. Poetry will not do this if you follow the guide. It will just silently use its own environment.

You can change the cache-dir value by editing the Poetry configuration.

I have not tried this.

Additionally, you can use the virtualenvs.in-project configuration variable to create virtual environments within your project directory.

I have not tried this either, but from what I can see in the related issues, this is also wrong.

At this point I started googling why I couldn't find any virtualenvs created by poetry.

Only after that, does the documentation mention that poetry will prefer an already active virtualenv. This might clear things up if you already have some experience or you have googled the issue. But for a first time user this guide is misleading and unusable.

[radoering]

@maflAT I understand the confusion if you only read the red box. However, if you read further it does not matter which method you choose (pipx, official installer, manual): all methods make sure that poetry's own virtual environment is not activated. Nevertheless, a PR improving the docs (maybe an additional warning that poetry's own environment should not be activated) is always welcome.

[aesteve]

I think the problem does not surface when poetry is installed but at a later point in time when used. So I agree, all the methods mentioned here do not activate the environment. But when you actually make use of poetry, say, to run poetry install then you shouldn't do that from poetry's virtual environment although it's really tempting to. And actually, the fact the commands mention poetry install only could let you think you have activated the virtual environment in which you installed poetry (I actually thought so and was really puzzled by poetry install not creating the .venv in my sub-directory although virtualenvs.in-project was set to true.

I tried to update the warning sign in #8833

[drhagen]

With the arrival of Pixi, this has become more of a problem. Pixi does not have a base environment by default. If you create a base environment in which to install Python and Poetry, you need to know to call it "base" or else Poetry will not work.

I would recommend this as the solution: Option called virtualenvs.ignore-conda-envs. If true, Poetry behaves as if it does not see the conda environment at all. If false, Poetry always installs into a conda environment when inside one. If a list of strings, Poetry ignores only those environments. The default is ["base"] keeping the current behavior unchanged.

[bmitc]

I just came across this issue upon installing a development environment based only on asdf, pipx, and poetry on a machine that had conda used on it previously for other projects.

I set the setting:

poetry config virtualenvs.in-project true

immediately after installing Poetry, so that no matter what, the Poetry usage would be confined to this new repository using this setup.

However, after running poetry install, and to my surprise, no .venv folder was created in the project root directory. It wasn't even created in the poetry config virtualenvs.path directory. Instead, it went and updated an existing conda virtual environment, and it did so without any confirmation. The only way I figured it out was running poetry env info --path. So now Poetry has installed and upgraded the very environment I didn't want it to touch.

To me, this is a bug and not a feature request. Poetry didn't do anything it was supposed to do as specified in its configuration, and it instead did the very thing I did not want it to do with no confirmation prior to doing so.

[austinmw]

over 3 years later and still no updates??

"Poetry should always be installed in a dedicated virtual environment to isolate it from the rest of your system. It should in no case be installed in the environment of the project that is to be managed by Poetry. "

Okay, so I create a dedicated virtual environment for poetry. Now, as per the guidance above, I try to tell poetry to use a different virtual env for project installation by running poetry config virtualenvs.in-project true.

What does poetry do? It completely ignores it's own recommendations and instead installs the project in the same virtual environment that poetry itself is installed in.

Is anyone else utterly confused by this behavior?? 🤦🏻‍♂️

[radoering]

Did you pay attention to the last sentence of the recommendation?

In addition, the isolated virtual environment in which poetry is installed should not be activated for running poetry commands.

[bmitc]

Did you pay attention to the last sentence of the recommendation?

@radoering Who and what comment are you responding to?

---

Also, as a general comment on this issue, I see that it's still labeled as a feature request and that there's some mention above about improving the documentation, but as I described above, this is a bug.

---

For those coming to this issue, a workaround does exist to force Poetry to ignore any conda environment. Here are the instructions that I have written in a README.md of mine to workaround this issue such that Poetry behaves as expected, as documented, and as configured.

• Install the poetry-conda Poetry plugin:

  python -m pip install --user poetry-conda
  

  This is required to ensure that Poetry does not use any environment belonging to conda.

• Set global Poetry settings:

  poetry config virtualenvs.create true
  poetry config virtualenvs.in-project true
  poetry config virtualenvs.ignore-conda-env true
  

  This configures Poetry to always create a virtual environment in a .venv directory in the Poetry project root directory and to ignore any conda environment. Important: This must be done prior to running poetry install, or you risk installing and upgrading packages in a conda environment if one is present.

[radoering]

Who and what comment are you responding to?

I responded to the comment directly above my comment but it might also apply to your comment. Respecting/using activated virtual environments is a feature even though it might look like a bug for some users. A perfectly valid workflow is to create a virtual environment without the help of Poetry, activate it and thereby let Poetry use it.

Also, as a general comment on this issue, I see that it's still labeled as a feature request [...] but as I described above, this is a bug.

Considering the title and the description of this issue I think this issue is a feature request.

[mforbes]

@radoering I think that part of the issue some users have is that the instructions say - install poetry in a dedicated virtual environment, but don't activate that environment. That is not how many people expect virtual environments to work (at least with conda – Pipx makes things better, but condax still has some issues). While one can manipulate paths to use the tools installed in an isolated environment, this can cause issues if not done carefully (which is the point of activating functional conda environments).

While using an active virtual environment is indeed a feature, it is strange and confusing that there is no robust way to turn off this feature when it causes issues.

Perhaps poetry could refuse to update the environment in which it is installed, and emit a helpful warning at that point of use with appropriate instructions? Maybe something like:

It appears you running poetry from an activated virtual environment. Any updates will be applied to this active environment, which is probably not what you want. As discussed in the documentation, poetry should not be run from an activated environment, but should instead be installed in a a dedicated virtual environment that is not active (but whose bin/ directory is on your PATH).

Are you sure you want to proceed? (y/N)

[bmitc]

Considering the title and the description of this issue I think this issue is a feature request.

With the following settings, it's expected that Poetry goes and uses another virtual environment without warning?

poetry config virtualenvs.create true                # This is the default value
poetry config virtualenvs.in-project true            # The default is false
poetry config virtualenvs.prefer-active-python false # This is the default value

It is very difficult, if not impossible, for Poetry users to be able to read the official documentation, configure settings, and then be aware of what Poetry will do. So I definitely agree that there is an opportunity to improve the consistency of the documentation, which is still a "bug" in my view, but I am still confused by the behavior of some of these settings. I'd be happy to potentially take a look at doing that improvement. But it is still unclear to me what the new feature being described here would be because it seems like the feature already exists in the above settings.

I do see the following note in the documentation for virtualenvs.create that says:

If Poetry detects it’s running within an activated virtual environment, it will never create a new virtual environment, regardless of the value set for virtualenvs.create.

In that specific note's sense, I suppose there is a feature request here, but it is confusing as to why that is the default behavior and why the above settings do not override it. So I think the bug here is that the settings and documentation and interpretation of the settings by Poetry is not consistent. So the improvement here could be:

1. Improve the consistency of the documentation and add more details of when settings do or do not apply.
2. Understand why the interpretation of the settings discussed in this issue still lead Poetry to use an activated environment. To me, this is still outstanding in terms of understanding this, at least from my point of view.
3. Add an explicit setting that configures whether Poetry uses an already activated environment or not, and default it to not using any activated environment. I still think such a feature request requires a better understanding of how the current settings are summed and interpreted by Poetry.

[renan-r-santos]

The current approach of Poetry treating only the Conda base environment differently

poetry/src/poetry/utils/env/env_manager.py

Lines 290 to 293 in 5bab98c

	conda_env_name = os.environ.get("CONDA_DEFAULT_ENV")
	# It's probably not a good idea to pollute Conda's global "base" env, since
	# most users have it activated all the time.
	in_venv = env_prefix is not None and conda_env_name != "base"

likely contributes to user confusion, even though there may be valid reasons for this behavior.
To enhance clarity and user control, I think it would be preferable to allow users to explicitly choose whether they want to reuse Conda environments or not. This opt-in/out approach would eliminate the need for Poetry to make assumptions about what users want, thereby providing a more transparent and predictable experience for users.

[radoering]

Perhaps poetry could refuse to update the environment in which it is installed, and emit a helpful warning at that point of use with appropriate instructions? Maybe something like: [...]

Might make sense but you have to consider that asking for user interaction is undesirable in CI.

It is very difficult, if not impossible, for Poetry users to be able to read the official documentation, configure settings, and then be aware of what Poetry will do.

I understand that the interaction/meaning of the settings can be confusing. Currently, I do not know how to improve the situation. I can only try to explain the intention although I do not know if it will help or add any new information compared to the documentation:

• desired default behavior: always use an activated virtual environment
  This allows you to create a virtual environment to your likings without Poetry and just use it by activating it. If you do not create a virtual environment by yourself and activate it, Poetry does it for you.
• this default behavior cannot be changed at the moment (this feature request is about an option to change this default behavior)
• virtualenvs.create defines what to do if no virtual environment has been activated and no environment has been created yet: Create a new virtual environment or use the system environement. The latter is strongly discouraged in most cases.
• virtualenvs.in-project defines where to create a virtual environment if no virtual environment has been activated and no environment has been created yet: Create the environment in a .venv folder in the project or in the cache directory.
• virtualenvs.prefer-active-python defines which Python to use if a virtual environment has to be created.
• virtualenvs.in-project and virtualenvs.prefer-active-python have no influence on whether a virtual environment is created or not. They only change where/how it is created.

[austinmw]

Did you pay attention to the last sentence of the recommendation?

In addition, the isolated virtual environment in which poetry is installed should not be activated for running poetry commands.

pardon my ignorance, but how do you run poetry installed in a virtual environment without activating it? If this is the intended usage, then I would expect most people to be very confused by it.

[ndevenish]

Did you pay attention to the last sentence of the recommendation?

In addition, the isolated virtual environment in which poetry is installed should not be activated for running poetry commands.

pardon my ignorance, but how do you run poetry installed in a virtual environment without activating it?

You can call the console script created in the bin/ folder directly without activating. It hardcodes the correct interpreter executable directly.

Tools like pipx take advantage of this and only expose the packages console_scripts (so you don’t get the python of the last environment on PATH. I think there are several other tools that work similarly.

I don’t think this is the primary problem being raised in this issue, however.

[Turiok]

Using the same env for conda and poetry destroy my poetry installation in the conda+poetry env.

I have a conda env for COTS like gitleaks, trufflehog or openldap client and poetry.
Then I would like a poetry env for my python dependencies of my code.

I installed my conda env then poetry install the dependencies in the conda env too.
It bugs me but worked fine until now.

I wanted add a new poetry dependencies with poetry add <package>
And poetry uninstall the virtualenv dependency in conda+poetry env.
Then any poetry command failed because virtualenv module was not found.

I'm using poetry-conda plugin now to make it works. But I thing this feature should be in default installation.

[avaz]

For my use case would also be very helpful in having the ability to make poetry disregard Conda or any other kind of virtual environment.
I have a somewhat complex project made of many modules in a mono repo fashion. Each module has its own pyproject.toml and it's managed by poetry which means that all of them have their own environment and their libraries don't conflict with other modules libraries.

This works well but I use something like Conda to install other project level dependencies that aren't necessary python ones, like for instance some dynamic libraries, ODBC drivers, and so on. Conda fits well for this as it allows to have all project dependencies in the same 'context' while maintaining each modules dependencies private to itself.

On my environment.yml I have the python version required by the project as well poetry which helps to keep everything in one place and reproducible.

In summary, considering the diverse use cases, it is advisable to allow users to determine when to create a new environment or reuse an existing one. For the time being, I am utilizing the poetry-conda Poetry plugin to address my requirements. However, it would be beneficial to have this plugin natively supported.