Poetry should handle the version of python like it handles everything else

[mmr-crexi]

Right now, when I start a new project, I can choose a python version. Say I choose python ^3.12. Oh no! I just realized that triton and pytorch don't yet use python 3.12! No problem, I'll just downgrade my python version in my poetry pyproject.toml file. If I do that, then I get this error:

Current Python version (3.12) is not allowed by the project (3.11.5).

This is nonsense. Just downgrade the python version, treat it like any other library. I can solve the problem by deleting the directory and restarting the entire project, but that's just awful UX.

Poetry creates a virtual environment for use when it is first run, and then that python version appears to be locked for all time and I'm having a hell of a time changing it. When people on my team are off by third digit version numbers (3.11.5 vs 3.11.6), well, that means that each of us have created incompatible python versions and we can't run each other's code.

Please, please, please, please, just allow poetry to treat python like any other library, like it does when the project is first created.

[dimbleby]

what you suggest is just not possible: poetry doesn't get to choose the python version in the same way that it gets to choose the version of every requirement.

if you have a project that has set requires-python to an exact version of python then the solution is simple: don't do that!

[mmr-crexi]

Not possible? I'm not asking for reversing gravity or time travel, I'm asking for the ability to hard reset a poetry installation so that I can start entirely from scratch. That is, kill everything from the ground up, and start again with a fresh python installation before going to other libraries. I can by-hand wipe virtual environments that have been set aside in directories that are clearly not intended for user intervention, but even that doesn't clear the warning that the python version in my pyproject.toml doesn't match what was in the first iteration of the file.

The minor/minor version number problem arises when I have 3.11.6 and someone else on my team has 3.11.5, and then we say it should be version 3.11, and then when poetry lock files move around, that disagreement also causes error messages to be sprayed out about python version incompatibility.

This is only a problem because some of the libraries I want to use (triton, pytorch, etc) are not yet compatible with python 3.12, but Poetry wants to install 3.12, so I find myself having to jump through a huge series of hoops to downgrade to 3.11.

I see a lot of conversations around using pyenv, but frankly, I was under the impression that poetry is supposed to be a one-stop shop. If I'm going to use pyenv, then why not just use virtualenvs, pip, and a makefile that can run make clean to wipe the venv and start anew when I need to?

[dimbleby]

Not gonna happen.

it is hard to see through the exaggeration to what your actual problem is, but let's try

I can by-hand wipe virtual environments that have been set aside in directories that are clearly not intended for user intervention,

poetry env list and / or poetry env remove 3.12 is I suppose what you want

that doesn't clear the warning that the python version in my pyproject.toml doesn't match what was in the first iteration of the file.

I don't know what warning you mean

minor/minor version number problem arises when I have 3.11.6 and someone else on my team has 3.11.5

I don't recognise this either. eg poetry init -n will put (say) python = "^3.11" in your pyproject.toml. If you are deliberately adding patch version numbers - simply stop doing that

when poetry lock files move around, that disagreement also causes error messages to be sprayed out about python version incompatibility.

again I don't know what you mean

I find myself having to jump through a huge series of hoops

nonsense

I was under the impression that poetry is supposed to be a one-stop shop.

Not at all. Use it - or don't - in whatever way works for you. Personally I indeed prefer virtualenvwrapper to manage my virtual environments.

[dimbleby]

haha, in fact experimenting with this a little myself I see that you either omitted or ignored the sentence that comes after "Current Python version ... is not allowed ...":

Please change python executable via the "env use" command.

poetry is already telling you how it supports the situation that you are in

[mmr-crexi]

So let me run you through what I'm doing here.

poetry init
choose default python (^3.12)
add torch
That's it.

poetry install
oh, breaks on triton: Unable to find installation candidates for triton (2.1.0)

whoops, forgot that I need to go to 3.11. Edit pyproject.toml to "<3.12".

poetry install

Current Python version (3.12.0) is not allowed by the project (<3.12).
Please change python executable via the "env use" command.

This is the fundamental error. Just nuke and replace the python environment. No need for other commands here. If I had chose <3.12 in the first place, no need to futz around with python env use command.

But OK, lemme go ahead and do env use.

poetry env use 3.11

mark@mmr-laptop:~/src/sample2$ poetry install
Updating dependencies
Resolving dependencies... (0.0s)

The current project's Python requirement (<3.12) is not compatible with some of the required packages Python requirement:
  - torch requires Python >=3.8.0, so it will not be satisfied for Python <3.8.0

Because no versions of torch match >2.1.1,<3.0.0
 and torch (2.1.1) requires Python >=3.8.0, torch is forbidden.
So, because sample2 depends on torch (^2.1.1), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the `python` or `markers` properties
    
    For torch, a possible solution would be to set the `python` property to ">=3.8.0,<3.12"

OK, still a headache. Last time I checked, 3.11 is greater than 3.8 and less than 3.12, so the dependency manager should have just used the current environment. Lemme go ahead and edit to >=3.8.0,<3.12.

And it works.

There's a reason this is a feature request, and not a bug. That is that the above process is misleading; 3.11 fits the requirements specified in the error message and I got to that version through the env use command, but apparently env use is insufficient, since the dependency resolver isn't satisfied (apparently, I'm guessing about what's not actually satisfied here). So let's try setting the python property to "3.11", instead of ">=3.8.0,<3.12".

Current Python version (3.11.6) is not allowed by the project (3.11).
Please change python executable via the "env use" command.

When did I set that current python version? I thought I was setting the current python version in the pyproject.toml in the first place. Is it referring to system python? Why should that version of python even matter, shouldn't the whole point of poetry be reproducibility in other systems regardless of their system python versions?

Either way, that's why I have to set the third version number, because 3.11 isn't the same as 3.11.6. What about ~3.11?

Warning: poetry.lock is not consistent with pyproject.toml. You may be getting improper dependencies. Run `poetry lock [--no-update]` to fix it.

No dependencies to install or update

Notice that there is no error message or indicator to use the tilde notation for a version like 3.11. I guessed from the docs (https://python-poetry.org/docs/dependency-specification/). There's the warning I'm talking about. So let's run poetry lock [--no-update]:

No arguments expected for "lock" command, got "[--no-update]"

Oh, maybe the brackets are extra:
poetry lock --no-update

OK, there we go.

These are all little frustrations and annoyances that I don't have to deal with for any other package in the pyproject.toml, just python, from what I can tell. That's my feature request: treat installing python the same as installing any other package, downgrade it on command as specified in the file, and keep the python version that's being used separate from all the other system components.

So let's go back to the beginning. I'll start a new project (in a new folder, because somehow this folder is associated with a poetry virtualenv somewhere and I don't know how to change that) and chose 3.11 as my python version.

Generated file:

[tool.poetry.dependencies]
python = "3.11"
torch = "^2.1.1"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Results in:

mark@mmr-laptop:~/src/sample3$ poetry install
The currently activated Python version 3.12.0 is not supported by the project (3.11).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

So before, my python environment is 3.11.6, which makes some kind of sense, since that's my system python. Where did 3.12.0 come from? This is a new directory, and the last time I used the env command, it was using a version of 3.11.

Do my frustrations make some kind of sense here? And again, I'm trying to be respectful of people's time, since this isn't a breaking bug, just an annoyance (to me), hence why I'm asking for some kind of improvement here.

Not at all. Use it - or don't - in whatever way works for you. Personally I indeed prefer virtualenvwrapper to manage my virtual >environments.

I was under the impression that poetry already uses virtualenvs under the hood to manager environments, given messages like Creating virtualenv sample2-RtlTWBM--py3.12 in /home/mark/.cache/pypoetry/virtualenvs. So is the suggestion to use virtualenv managers on top of poetry? That seems redundant, what am I missing? Why not just allow for everything in the virtualenv to be nuked and recreated from scratch?

[dimbleby]

You have taken three separate mis-steps and want poetry magically to fix them all - it can't happen.

First you declared that your project supported python 3.12, and tried installing a package that doesn't support python 3.12.

Then you declared that your project supported all pythons <3.12, and that was untrue for similar reasons.

Then you declared that your project supported exactly python 3.11.0 - this is what 3.11 means - and that wasn't what you intended either.

(You are not using the latest poetry; the error message now makes it a little clearer that when you give a python version you are declaring the range that your project supports.)

None of this is solved by recreating the environment; there is no magic wand. All of those things have meaning, just not what you wanted.

What finally fixed this for you is you providing a python version for your project that is compatible both with the packages you want to install, and the pythons you have available.

But that has to be on you; only you can say what python versions you intend your project to support. When you ask for incompatible packages and pythons, only you can say which of the incompatible things it is that you want to yield.

"I guessed from the docs" is a weird way to say "I read the docs and they were helpful". Nevertheless I'd suggest that the most likely way to salvage something productive from this would be to submit a docs merge request clarifying whatever part of this was unclear

[mmr-crexi]

I think what you're missing is that I messed up. I own that. I messed up multiple times.

What I'm asking for is a way to easily and quickly erase everything that I did. I don't see how to do that. Somehow, history is being kept, even when I'm not asking for it to be. Somehow, a default version of 3.12.0 is being chosen in a brand new project, despite the system library being set to 3.11.6 and there being nothing in the directory to begin with.

How can I start from scratch, erasing everything off of my machine? That's what I'm asking for. I don't want there to be floating virtual environments around there that I don't know how to find or manage, I want to know where they are. I've nuked that virtual env directory before, but still, there appear to be lingering vestiges of code lying about the place. Where are they, and how do I get rid of them? I can't imagine that once I've installed poetry once, I'm stuck with a bunch of unchangeable hidden defaults, but that seems to be the case with that 3.12.0 error message from my previous post.

And you also didn't really answer my question-- you're using virtualenvs to manage your poetry installations, but poetry itself uses virtualenv? That seems duplicative at best; what don't I understand that would make you want to have multiple layers of virtualenv?

[dimbleby]

if you have a virtual environment already activated then poetry respects that and doesn't interfere. I prefer to use that, and manage my environments outside of poetry. Your mileage may vary.

if you are using poetry-managed environments, then manage them using poetry env. In particular, delete them all with poetry env remove --all. This already exists and is documented and so on.

[mmr-crexi]

So now, creating sample4:

[tool.poetry.dependencies]
python = "3.11"
pytorch = "^1.0.2"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

mark@mmr-laptop:~/src/sample4$ poetry install
The currently activated Python version 3.12.1 is not supported by the project (3.11).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

Why does poetry think it should be using 3.12.1?

OK, lemme remove all:

mark@mmr-laptop:~/src/sample4$ poetry env remove --all
mark@mmr-laptop:~/src/sample4$ poetry install
The currently activated Python version 3.12.1 is not supported by the project (3.11).
Trying to find and use a compatible version. 

Poetry was unable to find a compatible version. If you have one, you can explicitly use it via the "env use" command.

No joy.

Changing the python version to ~3.11 from 3.11:

mark@mmr-laptop:~/src/sample4$ poetry install
The currently activated Python version 3.12.1 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.6)

Oh, so maybe now that I have an environment, it'll switch from using system python to the one I've just specified and it seems to have accepted, 3.11.6:

mark@mmr-laptop:~/src/sample4$ poetry lock
The currently activated Python version 3.12.1 is not supported by the project (~3.11).
Trying to find and use a compatible version. 
Using python3 (3.11.6)
Updating dependencies
Resolving dependencies... (0.3s)

Nope, still using some other Python version.

How is that happening? Is there any way to reset that? Why would it not be using the python version I specified in the pyproject.toml?

[dimbleby]

Why does poetry think it should be using 3.12.1?

Read more carefully. That is the opposite of what the error message says. The problem is that your currently active python is 3.12.1 and that poetry wants not to use that.

Changing the python version to ~3.11 from 3.11:

we already went through this. 3.11 is the same thing as 3.11.0, you do not have a python 3.11.0.

$ poetry lock
The currently activated Python version 3.12.1 is not supported by the project (~3.11).

this is the first time you have reported that one! This one I think is somewhere between an annoyance and a bug: poetry lock does not activate the project environment, but does complain if the current environment is incompatible.

If you want to raise a bug report covering exactly that, that would make sense.

[mmr-crexi]

OK, I'll open that bug.

For the 3.11 vs ~3.11, that seems like the same bug, no? If it doesn't have 3.11.0, shouldn't it just go get it and use that? What I'm saying is, shouldn't the very first thing that poetry does is to switch into a poetry-managed virtualenv and use that version of python for everything else? I get that my system python is whatever it is, but if poetry can run with it, then it does, but then it'd switch, no?

[dimbleby]

Poetry will only use pythons that it can find already on your computer.

Taking on responsibility for finding and installing a python 3.11.0 is definitely too much.

[DbCrWk]

tl;dr: if poetry absorbed pyenv, I would be very happy.

I want to chime in here because I enjoy using poetry, but I also trip up on this issue sometimes. Here's my understanding:

There is a difference between python as a dependency and python as an interpreter, which intersects with the difference between using a project as a library vs. using it as an application. If I am importing my project as a library, then setting the correct python version (or range of versions) in pyproject.toml is sufficient. However, if I want to actually run some files in my project as an application, i.e. invoking a file as a script with a python interpreter, then poetry assumes that you will figure out how to get the correct python interpreter.

In my mind, I assumed that setting the python version in the pyproject.toml file would be sufficient to solve both of these use cases. However, this is incorrect. Instead, I use pyenv to get a correct python interpreter and then ask poetry to use said interpreter when setting up a virtual environment. This interpreter has to satisfy the constraints set in the pyproject.toml file about python if there are any set, but is otherwise an orthogonal problem.

I think what @mmr-crexi is getting at is that this distinction is a bit subtle, especially since our mental model for other dependencies is that we can use poetry add and it appears for us to use in the virtual environment. But adding python as a dependency doesn't do that: it's the only one where we have to go outside the walled garden.

It may not be feasible or on the roadmap, but I will say that, as a feature, it would be awesome if poetry absorbed my use case of pyenv and that it could also automatically download and correctly setup python interpreters for me. From a users perspective, poetry init could prompt for a python version that you want to use, download that version (or use a cached one if already downloaded locally), and then create a virtual environment with it.

Perhaps this could be accomplished with a plugin?

[DbCrWk]

I will say that, either way, it might be helpful to add some additional information on Managing environments or in the FAQ to clarify how the environment python and setting the python version work and don't work with each other.

[dimbleby]

Pull requests to docs should be easy, if you have improvements please go ahead and make them.

The state of this issue tracker should make it obvious that poetry's surface area is already larger than the small number of maintainers and contributors can keep up with. Trying to bepyenv, on top of everything else, will not help.

pyenvalready exists - just use it!

[principalideal0]

I get that the OP is being a bit colorful about his frustrations, however they are common in other forms. For example, a contributor to the project (who presumably knows it better than the casual user) posted in 2020 about how this aspect of Poetry breaks automations for many folks: #1945

The proposed solution in that case doesn't work in mine at all, though it has similar characteristics. And the stickiness is directly tied to the special treatment Python versions get. In that thread as well, multiple users had no problem with managing the download and install of Python versions themselves.

I get it's a big project, and we all appreciate the folks who work hard on it. Thank you. It may just be the price of success for building something really amazing that folks want Poetry to take full control of asserting which environment to use. I think it's clear this is a significant issue though, since it's tied to a host of related challenges folks have, like this: #4055

Maybe a flag to make Poetry assertive when its invoked would knock out a fair number of usability challenges folks are experiencing? It would also not nuke existing setups / be backward compatible.

[jsimonlane]

To quickly chime in here, @DbCrWk hit the nail through the head here in his above post.

I didn't properly understand the distinction between Poetry as a library vs as a runtime interpreter until reading that post, but it makes a lot more sense now. The docs would benefit from making this more immediately clear.

[dimbleby]

Pull requests to docs should be easy, if you have improvements please go ahead and make them.

[DbCrWk]

@dimbleby I opened a PR #9256 for documentation.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.