SolveFailure with experimental installer and editable dependency

[petergaultney]

• Poetry version: 1.4.0
• Python version: various, including 3.7, 3.8, and 3.11
• OS version and name: MacOS 12 and 13
• pyproject.toml:

[tool.poetry]
name = "thds.foobar"
version = "2.1"
description = "FooBar"
authors = ["foobar"]
readme = "README.md"
packages = [{include = "thds", from="src"}]
include = ['py.typed', 'meta.json']

[tool.poetry.dependencies]
python = "^3.8"
typing-extensions = ">=3.8"
thds-core = {path = "../core", develop = true}

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

 Updating types-setuptools (67.5.0.0 -> 67.4.0.3)
  • Installing uri-template (1.2.0): Skipped for the following reason: Already installed
  • Installing urllib3 (1.26.14): Skipped for the following reason: Already installed
  • Installing wcwidth (0.2.6): Skipped for the following reason: Already installed
  • Installing webcolors (1.12): Skipped for the following reason: Already installed
  • Installing webencodings (0.5.1): Skipped for the following reason: Already installed
  • Installing websocket-client (1.5.1): Skipped for the following reason: Already installed
  • Installing widgetsnbextension (4.0.5): Skipped for the following reason: Already installed
  • Installing wrapt (1.15.0): Skipped for the following reason: Already installed
  • Installing xlrd (2.0.1): Skipped for the following reason: Already installed
  • Installing yarl (1.8.2): Skipped for the following reason: Already installed
  • Installing zipp (3.15.0): Skipped for the following reason: Already installed
  • Installing thds-adls (1.0 /Users/peter/repos/ds-monorepo/libs/adls): Failed

  SolveFailure

  Because -root- depends on thds.core (*) which doesn't match any versions, version solving failed.

  at ~/software/poetry/venv/lib/python3.8/site-packages/poetry/mixology/version_solver.py:349 in _resolve_conflict
      345│             )
      346│             self._log(f'! which is caused by "{most_recent_satisfier.cause}"')
      347│             self._log(f"! thus: {incompatibility}")
      348│
    → 349│         raise SolveFailure(incompatibility)
      350│
      351│     def _choose_package_version(self) -> str | None:
      352│         """
      353│         Tries to select a version of a required package.

The following error occurred when trying to handle this error:


  SolverProblemError

  Because -root- depends on thds.core (*) which doesn't match any versions, version solving failed.

  at ~/software/poetry/venv/lib/python3.8/site-packages/poetry/puzzle/solver.py:163 in _solve
      159│             packages = result.packages
      160│         except OverrideNeeded as e:
      161│             return self._solve_in_compatibility_mode(e.overrides)
      162│         except SolveFailure as e:
    → 163│             raise SolverProblemError(e)
      164│
      165│         combined_nodes = depth_first_search(PackageNode(self._package, packages))
      166│         results = dict(aggregate_package_nodes(nodes) for nodes in combined_nodes)
      167│

Issue

Hi! Using poetry config experimental.new-installer false makes this go away - all of our developers have been on 1.3.2 until recently, but a few upgraded and our monorepo does not work for them anymore.

Downgrading is one workaround, but the easier one is just disabling 'no longer experimental' installer which, unfortunately, still seems somewhat experimental to us.

I don't really have any more information about this that I know to be relevant, but I'll try to answer any questions you have.

[dimbleby]

please provide a way to reproduce: a sample git repository would work, or a docker container is an approach some folk prefer.

[petergaultney]

to be painfully honest, I don't think it's likely that I'll have time for that any time soon. I realize this is the curse of the open source maintainer, to get lots of bugs filed and not enough info to reproduce. :( Hopefully this bug sitting open will attract other people who find themselves having the same issue, or maybe I'll find an extra hour or two to play around with setting up other git repos.

All our users will just end up using the internal (downgrade/disable new installer) workaround for the time being.

[dimbleby]

probably best just close this then.

fwiw I took your pyproject.toml fragment and invented a repository around it and it worked fine. So there's definitely not enough to go on here and no prospect of anything happening without further input.

[wagnerluis1982]

This is strikingly similar to the bug I found hidden in one installer test, see #7498 (comment)

[dimbleby]

there are approximately 100 issues that look like this in the sense that they're a SolveFailure, I don't (yet) see any particular reason to connect these two.

[wagnerluis1982]

Well, I say is similar because they happen in the same line numbers with almost the same error message Because -root- depends on <dependency> (*).

So, I think the installer was unintentionally released with this bug, relying on the "valid" test results. Bringing here the discussion, the following test is invalid and has very similar error, only change dependency name, and keeping the (*).

But why is the test passing? For the following reasons:

1. The test is not checking installer output. If checked, it would find the error.
2. Lock file is written even though installer fails (known defect If installation of a dependency fails, it remains in the lock file until it is updated with poetry lock #395).

poetry/tests/installation/test_installer.py

Lines 1147 to 1166 in 3e32a4f

	def test_installer_with_pypi_repository(
	package: ProjectPackage,
	locker: Locker,
	installed: CustomInstalledRepository,
	config: Config,
	env: NullEnv,
	):
	pool = RepositoryPool()
	pool.add_repository(MockRepository())
	installer = Installer(
	NullIO(), env, package, locker, pool, config, installed=installed
	)
	package.add_dependency(Factory.create_dependency("pytest", "^3.5", groups=["dev"]))
	installer.run()
	expected = fixture("with-pypi-repository")
	assert expected == locker.written_data

[dimbleby]

this discussion belongs in #7498: if and when you understand what's going on then we can try and decide whether it has any connection to the issue here.

[wagnerluis1982]

I understand your reasoning, but I have to disagree.

This is way bigger than #7498, which only uncovered the issue (rather than introduce it).

[dimbleby]

afaik we (certainly I but I think also you) don't have any idea what's going on in either #7498 or here

• in Only write lock file when installation is success #7498 because no-one has debugged it
• here because OP is unwilling or unable to provide anyone with the means to debug it

speculating that they are the same is all very well, but please deal with #7498 in #7498.

if understanding that brings some insight here, super.

[dimbleby]

they're very much not the same thing but investigating did bring insight, so we'll call it a score draw ;-)

pretty sure this is a duplicate of #7574, please close

[wagnerluis1982]

afaik we (certainly I but I think also you) don't have any idea what's going on in either #7498 or here

You're right, I have no idea 🤣

if understanding that brings some insight here, super.if understanding that brings some insight here, super.

Agreed :-)

[gandersen101]

I am one of @petergaultney's coworkers. I can try to build a toy repo to reproduce this if needed. Give me a day or two.

[gandersen101]

In the process of recreating this I believe I've isolated what caused our error. Whether or not you all consider this an error, I'll leave for you all to decide/explain.

We specify an internal package (thds.core from the error message in the original issue post) hosted on a private Artifactory server, as a build-system.requires in many of our pyproject.toml files. This allows us to perform some additional build scripting. Presumably due to the changes in #6205, the poetry back-end no longer looks at a pip.conf for extra-index-url entries when installing build-system.requires?

My team and I can work around this, so it isn't a blocker for us, but something for the poetry team to be aware of if you are not already.

If you all decide to close this that's fine. Or if you decide this is an issue you all want to solve that's fine too. Thanks!

[dimbleby]

with the modern installer, poetry should respect sources defined in pyproject.toml even for build requirements.

that it previously didn't was #3249, it sounds as though you've found the workaround for that and gotten so used to it that you're now not prepared for the fixed behaviour!

[petergaultney]

we don't want to specify the build source in the pyproject.toml, because then poetry falls back to the known slow behavior with accessing non-default, secondary package indexes. This leads to a factor of ten increase in lock times, which isn't sustainable (and actually one of the main reasons we switched to poetry is that it handles lock times better than the competition because of its optimizations around PyPI).

Unless the new installer somehow magically solves the slow lock times with other indexes enabled?

[dimbleby]

the modern installer does not use pip. relying on pip configuration to influence poetry behaviour was always an ugly workaround: poetry is not pip,

so it sounds as though this is completely understood, there will be no changes, we should close this issue down

if you believe you have unusual or interesting requirements re secondary package indexes - and I would say that wanting to use them only for build requirements counts as unusual - then you might want to chip in on #6713. I'm fairly sure that's a use case that no-one has considered. Possibly it's so weird that you won't get any traction anyway, but you don't know unless you ask!

[dimbleby]

this issue drifted and eventually settled on "we know how to fix this but we don't like to use secondary repositories because slow".

That should be addressed in 1.5.0 by supplemental (or explicit) repositories.

This can be closed.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.