Resolution fails for boto3/botocore in multiple python versions

[mcouthon]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: macOS Catalina 10.15.7
• Poetry version: Poetry version 1.1.4
• Link of a Gist with the contents of your pyproject.toml file: link

Issue

We need very specific boto3/botocore versions for compatibility with legacy code. In our non-poetry projects we set the following in the requirements.txt:

botocore==1.12.192;python_version<"3.0"
botocore==1.17.63;python_version>"3.0"
boto3==1.9.192;python_version<"3.0"
boto3==1.14.63;python_version>"3.0"

And installation works great in both python 2.7 and python 3.x.

I've attempted to create a comparable pyproject.toml:

[tool.poetry.dependencies]
python = "^2.7 || ^3.6"

botocore = [
    {version = "1.12.192", python = "^2"},
    {version = "1.17.63", python = "^3"}
]
boto3 = [
    {version = "1.9.192", python = "^2"},
    {version = "1.14.63", python = "^3"}
]

But poetry lock results in an error (on both python versions). Here's the error in a python 3.8 env:

   1: conflict: boto3 (1.14.63) depends on botocore (>=1.17.63,<1.18.0)
   1: ! not botocore (>=1.17.63,<1.18.0) is satisfied by botocore (1.12.192)
   1: ! which is caused by "test depends on botocore (1.12.192)"
   1: ! thus: boto3 is forbidden
   1: ! boto3 (1.14.63) is satisfied by boto3 (1.14.63)
   1: ! which is caused by "test depends on boto3 (1.14.63)"
   1: ! thus: version solving failed

And while the resolution starts with:

   0: Retrying dependency resolution with the following overrides ({Package('test', '0.1.0'): {'botocore': <Dependency botocore (1.12.192)>, 'boto3': <Dependency boto3 (1.9.192)>}}).

It at some point switches to:

   0: Retrying dependency resolution with the following overrides ({Package('test', '0.1.0'): {'botocore': <Dependency botocore (1.12.192)>, 'boto3': <Dependency boto3 (1.14.63)>}}).

Which of course causes the error (as these two are indeed incompatible versions).

Full output of poetry lock -vvv can be seen here.

[tchar]

Not sure if I should create a new issue, but this looks similar to mine.

I have a similar problem with dev dependencies.

[tool.poetry.dependencies]
python = ">=2.7"
six = [
    # Tox needs this version
    {version = "^1.14.0", python = ">=2.7 !=3.0.* !=3.1.* !=3.2.*"},
     # Python 3.0 through 3.2 need this version
    {version = "==1.11.0", python = ">=3.0.* <=3.2.*"}
]

[tool.poetry.dev-dependencies]
# Tox should not be installed on python versions where 
# six==1.11.0 is installed (3.0, 3.1, 3.2) plus (3.3, 3.4) which are not supported by tox.
tox = { version = "^3.24.1", python = ">=2.7 !=3.0.* !=3.1.* !=3.2.* !=3.3.* !=3.4.*" }

Output

  SolverProblemError

  Because no versions of tox match >3.24.1,<4.0.0
   and tox (3.24.1) depends on six (>=1.14.0), tox (>=3.24.1,<4.0.0) requires six (>=1.14.0).
  So, because yourmodule depends on both six (==1.11.0) and tox (^3.24.1), version solving failed.

I forced uploaded to PyPI and I can normally install my package through pip (it just mentions that the dev dependency is not getting installed, which end users don't really care unless they want to test)

Also I am telling poetry not to install tox on a superset of python versions that I specified for six==1.11.0, so this should't even be a conflict.

[nugend]

Anyone have any ideas on how to work around this issue?

[mcouthon]

We just gave up, and created a python2 incompatible version of the package 🤷

[radoering]

I think, I found a solution for this issue. With the changes in PR #4695, I was able to lock the boto3/botocore example. The solution relies on poetry recognizing if the intersection of markers is empty. Unfortunately, there is also some room for improvement. See #4694 for details. Without this improvement to poetry-core, some unnecessary combinations of overrides are solved in dependency resolution, resulting in an addtional version of botocore in the lock file. I am not 100 % sure, but I think this additional entry in the lock file does no damage, because it not used for any python version.

[GtTmy]

I have a similar case. poetry (1.1.12) cannot solve the following dependencies.
And I confirmed that #4695 makes it possible.

[tool.poetry.dependencies]
python = "^2.7 || ^3.7"

[tool.poetry.dev-dependencies]
pytest = [
    {version="^4.6.11", python="^2.7"},
    {version="^6.2.0", python="^3.7"}
]
pytest-mock = [
    {version="^1.13.0", python="^2.7"},
    {version="^3.6.1", python="^3.7"}
]

With poetry 1.1.12

https://gist.github.com/GtTmy/22bfd306a53ed7538eededd31f443cca

With #4695

$ python -m poetry update
Updating dependencies
Resolving dependencies... (14.0s)

Writing lock file

Package operations: 9 installs, 0 updates, 0 removals

  • Installing pyparsing (3.0.6)
  • Installing attrs (21.2.0)
  • Installing iniconfig (1.1.1)
  • Installing packaging (21.3)
  • Installing pluggy (1.0.0)
  • Installing py (1.11.0)
  • Installing toml (0.10.2)
  • Installing pytest (6.2.5)
  • Installing pytest-mock (3.6.1)

[GrimHacker]

I believe i'm seeing this as well.
And I confirmed that #4695 makes it possible.

├── grimlib
│   └── __init__.py
├── poetry.lock
└── pyproject.toml

[tool.poetry]
name = "grimlib"
version = "1.0.0" 
description = "poetry test"
authors = ["GrimHacker"]

[tool.poetry.dependencies]
python = "^2.7 || ^3.7"

# For py3.10 we need to use pytest >=6.2.5 (https://github.com/pytest-dev/pytest/issues/8539)
# For pytest >= 6.1.0 we need to use pytest-rerunfailures >=9.1.1 (https://github.com/pytest-dev/pytest-rerunfailures/issues/128)
[tool.poetry.dev-dependencies]
pytest = [
    { version = "<=4.6.11", python = "~2.7" },  # dropped py2 in 5.0
    { version = "<8", python = ">=3.7,<3.10" },
    { version = ">=6.2.5,<8", python = "^3.10" }  # >=6.2.5 https://github.com/pytest-dev/pytest/issues/8539
]
pytest-rerunfailures = [
    { version = "<9.0", python = "~2.7" },  # dropped py2 in 9.0
    { version = "<11", python = ">=3.7,<3.10" },
    { version = ">=9.1.1,<11", python = "^3.10"}  # For pytest >= 6.1.0 we need to use pytest-rerunfailures >=9.1.1 (https://github.com/pytest-dev/pytest-rerunfailures/issues/128)
]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

With poetry 1.1.13

$ poetry lock -vvv
...
  SolverProblemError

  Because no versions of pytest-rerunfailures match >9.1.1,<10.0 || >10.0,<10.1 || >10.1,<10.2 || >10.2,<11
   and pytest-rerunfailures (9.1.1) depends on pytest (>=5.0), pytest-rerunfailures (>=9.1.1,<10.0 || >10.0,<10.1 || >10.1,<10.2 || >10.2,<11) requires pytest (>=5.0).
  And because pytest-rerunfailures (10.0) depends on pytest (>=5.3), pytest-rerunfailures (>=9.1.1,<10.1 || >10.1,<10.2 || >10.2,<11) requires pytest (>=5.0).
  And because pytest-rerunfailures (10.1) depends on pytest (>=5.3)
   and pytest-rerunfailures (10.2) depends on pytest (>=5.3), pytest-rerunfailures (>=9.1.1,<11) requires pytest (>=5.0).
  So, because grimlib depends on both pytest (<=4.6.11) and pytest-rerunfailures (>=9.1.1,<11), version solving failed.

With #4695

$ poetry lock -vvv
...
   0: Resolved with overrides: ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}, Package('pytest', '4.6.11'): {'more-itertools': <Dependency more-itertools (>=4.0.0,<6.0.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<=4.6.11)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<9.0)>}, Package('pytest', '4.6.11'): {'more-itertools': <Dependency more-itertools (>=4.0.0)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<8)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<8)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (>=6.2.5,<8)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (>=6.2.5,<8)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (>=9.1.1,<11)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<empty>)>}}), ({Package('grimlib', '1.0.0'): {'pytest': <Dependency pytest (<empty>)>, 'pytest-rerunfailures': <Dependency pytest-rerunfailures (<empty>)>}})

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.