Private Repository – Extras suddenly failing to match

[SamuelMS]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Mac OS Catalina (10.15.5)
• Poetry version: 1.0.9
• Link of a Gist with the contents of your pyproject.toml file:

Issue

I'm encountering a bizarre, frustrating issue when using a package in a private repository that also provides python "extras" options. There are two packages at play here (names de-identified): core and api. The api package depends on the core package.

Previously, the core package was built with the following (simplified) setup.py script:

install_requirements = ['pandas==0.24.2', 'numpy==1.16.4', 'requests==2.23.0', 'ujson==1.35', ]
setup_requirements = []
test_requirements = ['pytest-runner==5.2', 'pytest==5.4.1', ]
extra_requirements = {
    'aws': ['s3fs==0.1.5', ],
    'excel': ['XlsxWriter==1.1.1', 'xlrd==1.2.0', ],
    'performance': ['simplejson==3.16.0', ],
    'full': [],
}

extra_requirements['full'] = list(set(itertools.chain.from_iterable(extra_requirements.values())))

setup(
    # <snip>
    name='core',
    python_requires='>=3, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, <4',
    install_requires=install_requirements,
    extras_require=extra_requirements,
    license="proprietary",
    include_package_data=True,
    setup_requires=setup_requirements,
    tests_require=test_requirements,
    version='1.0.0',
    # <snip>
)

When we migrated to poetry, the core library shipped with the following (simplified) pyproject.toml:

[tool.poetry]
name = "core"
version = "2.0.0"
# <snip>

[tool.poetry.dependencies]
python = "^3.7"

pandas = "0.24.2"
numpy = "1.16.4"
requests = "2.23.0"
ujson = "1.35"

s3fs = {version = "0.1.5", optional = true}
boto3 = {version = "1.14.1", optional = true}
XlsxWriter = {version = "1.1.1", optional = true}
xlrd = {version = "1.2.0", optional = true}
simplejson = {version = "3.16.0", optional = true}

[tool.poetry.extras]
aws = ["s3fs", "boto3"]
excel = ["XlsxWriter", "xlrd"]
performance = ["simplejson"]
full = ["s3fs", "XlsxWriter", "xlrd", "simplejson"]

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

For simplicity, let's say core went from 1.0.0 to 2.0.0 with the poetry changes.

Next, the api package attempts to install the core package via its pyproject.toml:

[tool.poetry]
name = "api"
# <snip>

[tool.poetry.dependencies]
python = "^3.7"

core = {version = "1.0.0", extras = ["full"]}

[[tool.poetry.source]]
name = "private"
url = "https://private-repository/pypi/core/simple/"

Running poetry update with core@1.0.0 works as expected, with the extra packages listed in the [full] requirements being properly installed:

Updating dependencies
Resolving dependencies... (15.6s)

Writing lock file


Package operations: 4 installs, 1 update, 0 removals

  - Installing s3fs (0.1.5)
  - Installing simplejson (3.16.0)
  - Installing xlrd (1.2.0)
  - Installing xlsxwriter (1.1.1)
  - Installing core (1.0.0)

However, changing the version tag to 2.0.0 – the version with poetry – causes some bizarre behavior. The first time I run poetry update with core@2.0.0, the [full] packages are removed from the lock file.

Updating dependencies
Resolving dependencies... (15.8s)

Writing lock file


Package operations: 0 installs, 1 update, 4 removals

  - Updating core (1.0.0 -> 2.0.0)
  - Removing s3fs (0.1.5)
  - Removing simplejson (3.16.0)
  - Removing xlrd (1.2.0)
  - Removing xlsxwriter (1.1.1)

The second (and subsequent) time I run poetry update (no changes), I get an error:

Updating dependencies
Resolving dependencies... (15.1s)

[SolverProblemError]
Because core (2.0.0) depends on xlrd (1.2.0) which doesn't match any versions, core is forbidden.
So, because api depends on core (2.0.0), version solving failed.

My guess is that poetry is trying to install the [full] extras from the private repository with 2.0.0 and fails to fall back to PyPI for some reason, which is why it fails to find it... although I don't know why it removes the extras in the first place.

I'm pretty stuck here, so I would appreciate some guidance.

[SamuelMS]

Additional note: pip itself is actually able to install the core[full]==2.0.0 package – with extras – when provided with pypi as an additional index.

pip install --extra-index-url https://pypi.org/simple core[full]==2.0.0

Installing collected packages: idna, chardet, urllib3, certifi, requests, numpy, pytz, six, python-dateutil, pandas, ujson, xlrd, XlsxWriter, jmespath, docutils, botocore, s3transfer, boto3, s3fs, simplejson, core
Successfully installed XlsxWriter-1.1.1 boto3-1.14.14 botocore-1.17.17 certifi-2020.6.20 chardet-3.0.4 core-2.0.0 docutils-0.15.2 idna-2.10 jmespath-0.10.0 numpy-1.16.4 pandas-0.24.2 python-dateutil-2.8.1 pytz-2020.1 requests-2.23.0 s3fs-0.1.5 s3transfer-0.3.3 simplejson-3.16.0 six-1.15.0 ujson-1.35 urllib3-1.25.9 xlrd-1.2.0

I think this means that poetry fails to provide PyPI as a backup index when installing another package that was built by poetry, so extras fail to install?

[SamuelMS]

@finswimmer Hey, hate to tag you here directly, but this is a blocker for adopting poetry for us. Would love to get some eyes on this, at least to confirm I'm not doing something dumb on our end.

[finswimmer]

Hey @SamuelMS , no problem to ping me :) But in this case I cannot help, as I'm not familiar with private repository. Maybe @stephsamson , @sdispater or @abn can have a closer look at this.

fin swimmer

[seansfkelley]

I'm seeing this issue too: private repository and dependency extras being both required by the solver, but also ignored by the solver. The issue seems to be related to overlapping extras, as we were able to solve it by taking extras blocks that looked like this:

[tool.poetry.extras]
server = [
  "sqlalchemy",
  "fastapi",
  "uvicorn",
]
executor = [
  "fastapi",
  "uvicorn",
  "typer",
]

and split their overlaps so it looked like this:

[tool.poetry.extras]
server = [
  "sqlalchemy",
]
executor = [
  "typer",
]
server_or_executor = [
  "fastapi",
  "uvicorn",
]

(This was a wild crapshoot guess -- I happened to notice that the two dependencies giving us issues were the only ones who appeared in the published PKG-INFO with the word "or", so at first I thought it was some kind of dependency specification syntax error.)

If you take this far enough, then every dependency will boil down to its own extra, which is really not great. That said, this workaround seems stable enough for the time being.

The issue is compounded by error reporting that is inconsistent and therefore misleading. An excerpt follows in heavily redacted form below (note that this is before the splitting of the overlaps described above):

(test) ➜  test poetry add my-private-dependency -E executor -vvv
Using virtualenv: /Users/<redacted>/.pyenv/versions/3.8.2/envs/test
PyPI: No packages found for my-private-dependency *
private-pypi: 8 packages found for my-private-dependency *
Using version ^0.8.0 for my-private-dependency

Updating dependencies
Resolving dependencies...
   1: fact: root-pkg is 2.0.0
   1: derived: root-pkg
   1: fact: root-pkg depends on my-private-dependency (^0.8.0)
   1: selecting root-pkg (2.0.0)
   1: derived: my-private-dependency (^0.8.0)
PyPI: No packages found for my-private-dependency >=0.8.0,<0.9.0
private-pypi: 1 packages found for my-private-dependency >=0.8.0,<0.9.0
PyPI: Getting info for my-private-dependency (0.8.0) from PyPI
   1: fact: my-private-dependency (0.8.0) depends on fastapi (>=0.54.1,<0.55.0)
   1: fact: my-private-dependency (0.8.0) depends on typer (>=0.3.0,<0.4.0)
   1: fact: my-private-dependency (0.8.0) depends on uvicorn (>=0.11.5,<0.12.0)
   1: selecting my-private-dependency (0.8.0)
   1: derived: uvicorn (>=0.11.5,<0.12.0)
   1: derived: typer (>=0.3.0,<0.4.0)
   1: derived: fastapi (>=0.54.1,<0.55.0)
PyPI: 2 packages found for uvicorn >=0.11.5,<0.12.0
private-pypi: 2 packages found for uvicorn >=0.11.5,<0.12.0
PyPI: 2 packages found for fastapi >=0.54.1,<0.55.0
private-pypi: 2 packages found for fastapi >=0.54.1,<0.55.0
   1: fact: typer (0.3.0) depends on click (>=7.1.1,<7.2.0)
   1: selecting typer (0.3.0)
   1: derived: click (>=7.1.1,<7.2.0)
   1: selecting uvicorn (0.11.6)
   1: selecting fastapi (0.54.2)
   1: Version solving took 1.684 seconds.
   1: Tried 1 solutions.

[SolverProblemError]
Because no versions of my-private-dependency match >0.8.0,<0.9.0
 and my-private-dependency (0.8.0) depends on uvicorn (>=0.11.5,<0.12.0), my-private-dependency (>=0.8.0,<0.9.0) requires uvicorn (>=0.11.5,<0.12.0).
So, because no versions of uvicorn match >=0.11.5,<0.12.0
 and root-pkg depends on my-private-dependency (^0.8.0), version solving failed.

Notice that it correctly reports that PyPI and the private PyPI instance both have two matching versions of uvicorn (which is true), and that it even later selects* what would be a compatible version of uvicorn (0.11.6), before complaining that there is no legal solution.

* I assume "select" means "attempt to solve for" rather than "decide is the correct solution".

[mdgilene]

I experience this same issue. I have worked around it for now by removing my "all" extra definition that includes all optional dependencies and left only the individual extras that don't have any overlap. Would be great to see this fixed as it is cumbersome doing "poetry install -E extra1 -E extra2 -E extra3 ... -E extraN" to install all dependencies. (Needed when developing a library and wanting to run tests).

[SamuelMS]

@stephsamson @sdispater @abn Per the note from @finswimmer above, is this something one of you can help us look into?

[bibz]

I experience the same problem than mentioned in #2494 but my search for an existing issue first landed me here. The issues symptoms look similar IMO.

[SamuelMS]

I experience the same problem than mentioned in #2494 but my search for an existing issue first landed me here. The issues symptoms look similar IMO.

Thanks for this. #2300 looks related.

[abn]

@SamuelMS happy to take a look into this, but before I do, can someone please make sure this is still an issue using current master? Also, please ensure that the lock file is regenerated too (should not have an impact here, but just for sanity).

[eino]

I have the same issue: poetry suddenly failing to install a lib with extras from a private pypi. Not sure about master but at least in 1.1.0b2 it is still present.
Edit: by the way, I'm also on Mac OS

[eino]

I solved the issue on my machine, it was a configuration issue. Even though the pyproject.toml was complete, I also also needed to configure the repository via poetry config repositories.my-repo url and poetry config http-basic.my-repo login... Not sure why it worked before.

[mgresko]

@abn Still seeing this issue

[maffka123]

Same here. I am trying to add this package, and get access forbidden every time. With pip it works:
pip install --extra-index-url=https://pypi.celonis.cloud/ pycelonis

[ronlut]

This happens to us as well. Any progress with that? Can't really manage extras with this behaviour

[SamuelMS]

Closing this as we went with a different solution.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.