Please remove cap on python version in pyproject.toml

[joelb123]

pandas-stubs's current upper bound on python version blocks every downstream package that also uses poetry from specifying python <4.

Note that this is a separate issue from actually supporting python 3.12; pandas 2.0 does not yet support python 3.12, so actual support of python 3.12 isn't possible at this time. But the persnickety solver that pandas uses means that downstream users like me will have to wait on you to uncap your python version. By that time, though, many of us will have moved on to other things, while our packages will stop building even though they may work perfectly well under 3.12.

This issue is extensively discussed in Henry Schreiner's blog post on poetry's decisions on capping. Unless you can affirmatively assert that pandas-stubs will not work on 3.12 at the time that pandas supports 3.12, you shouldn't dismiss all us downstream users in our optimism and force us to update our pyprojects.toml at some future time. As someone pointed out in a closed poetry issue on this topic, it took 4 years to force a much-used dependency (isort) to change a single character misreporting its compatibility.

Yes, this is really poetry's problem, not yours. Until they change things (and they have refused), the proper solution is to use a different build tool. But capping the python version contributes to a growing dependency hell.

[Dr-Irv]

Unless you can affirmatively assert that pandas-stubs will not work on 3.12 at the time that pandas supports 3.12

It is likely that when pandas supports version 3.12, there may be a period of time between pandas support of 3.12 and pandas-stubs support of 3.12, because there are possible deprecations in the python typing spec that are still undecided as to when they will be implemented.

While I can't affirmatively assert that pandas-stubs will not work on 3.12, I also can't affirmatively assert that pandas-stubs will work on 3.12 without testing it.

Because of this limitation of poetry, we could consider changing the build system. Would like the opinion of @twoertwein and @breno-jesus-fernandes on this topic.

[twoertwein]

pandas-stubs should work with any python version (zero runtime) - the important part is the type checker :) I would even be fine going with python = ">=3.8". I tested it locally and non of our poetry dependencies caused an issue with it.

[joelb123]

Yes, it's true that pandas-stubs presumably works with any python version which pandas supports. But having pandas capped at <3.12 causes the following problem for downstream projects which also use poetry:

$ poetry update # in myproject's context
Updating dependencies
Resolving dependencies... (0.6s)

The current project's Python requirement (>=3.10,<4) is not compatible with some of the required packages Python requirement:
  - pandas-stubs requires Python >=3.8,<3.12, so it will not be satisfied for Python >=3.12,<4

Because no versions of pandas-stubs match >1.5.2.221124,<1.5.2.221213 || >1.5.2.221213,<1.5.2.230105 || >1.5.2.230105,<1.5.3.230203 || >1.5.3.230203,<1.5.3.230214 || >1.5.3.230214,<1.5.3.230227 || >1.5.3.230227,<1.5.3.230304 || >1.5.3.230304,<1.5.3.230321 || >1.5.3.230321
 and pandas-stubs (1.5.3.230304) requires Python >=3.8,<3.12, pandas-stubs is forbidden.
So, because pandas-stubs (1.5.3.230321) requires Python >=3.8,<3.12
 and myproject depends on pandas-stubs (>=1.5.2.221124), version solving failed.

  • Check your dependencies Python requirement: The Python requirement can be specified via the "python" or "markers" properties
    
    For pandas-stubs, a possible solution would be to set the "python" property to ">=3.10,<3.12"

For some time, setting the downstream project's python cap to <4 would cause poetry's solver to select a really old version of pandas-stubs (along with pandas 1.1 IIRC) which had no such cap. Again, this is not your project's fault, but it is forcing us downstream to follow whatever you do, rather than let the solver do its work. For project that don't get much love (a lot of scientific packages, unfortunately) this results in an increasing number of packages that have to be forced-unpinned, which is just really icky from a rational packaging perspective.

Changing your build system to a less-opinionated and more standards-based solver would be great, if you are willing to change your developer workflow and CI. @henryiii has a packaging guide that suggests pdm using flit or hatchling as the build backend, along with the gotchas.

[breno-jesus-fernandes]

Hmm interings points ... We could for while follow the suggestion from @joelb123 and @twoertwein and just remove the cap, but after that we could migrate the dependency system to something like pdm which provides a flexible build system, simple to use and follow the peps.

[henryiii]

It is invalid to add an upper pin to the Requires-Python metadata slot, regardless of what backend you use. Regardless of what you support, the ecosystem simply isn't designed to allow this metadata slot to have an upper bound. There was a discussion about fixing this, but it would require the ability to detect upper caps, which would require quite a bit of new code in packaging. As it stands, it is only a feature that allows you to back solve for older versions of packages, and older versions never are more likely to support a newer version of Python.

If it's helpful, numba (which is the definition of a package that can't support unseen versions of Python) removed this cap, and is now getting much better failure messages.

Poetry's locking solver on downstream projects is usually the issue. Poetry's single setting for two distinct ideas (the filling of the metadata slot, which affects all downstream projects, and the bounds on the locking solve, which is only a local setting) is what generally causes this to be misconfigured, as you can't perform a locking solve without setting this to the lowest capped value and thereby ruining your package metadata.

[joelb123]

Ah, thanks, @henryiii. I have updated the title of the issue to reflect what I should have asked for.

Also thank to you stubbers for listening. This isn't an urgent issue for me, just a long-standing peeve that has risen to be above the level of the noise because the poetry crowd has thumbed their noses at the problem.

[henryiii]

I think you meant "cap on"

[twoertwein]

@henryiii has a packaging guide that suggests pdm using flit or hatchling as the build backend, along with the gotchas.

I will not reject if someone provides a PR with a maintainable and easy-to-use poetry alternative - but I must admit, I'm very happy with poetry overall. It is fast, ensures packages are compatible with each other's versions, and simplifies many workflows (from environment creation to publishing to pypi)! I much rather we use python = ">=3.8" in pyporject.toml than replacing poetry.

If we replace poetry, it might be good/consistent to pick whatever pandas uses (although I'm not a fan of conda).