Editable installs sometimes cannot be imported (poetry 1.1.0)

[ponas]

• I am on the latest Poetry version.

• I have searched the issues of this repo and believe that this is not a duplicate.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Fedora 32 (also reproduced with Docker base image python:3-slim)

• Poetry version: 1.1.1

• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/ponas/f02c8228a4e366bc255f4f4d1dd32cec

Issue

This issue first appears with Poetry 1.1.0. I cannot reproduce this issue on 1.0.9.

When Poetry installs the two packages from git, no exceptions occur, but sometimes one of them cannot be imported in the resulting environment:

$ poetry install
[snip]
  • Installing cerebrum-client (1.9.2 95a1839)
  • Installing tofh (2.0.6 8f9dac0)
$ poetry run python -c 'import tofh'
$ poetry run python -c 'import cerebrum_client'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'cerebrum_client'

I was unable to reproduce this by installing packages such as Django, requests and flask via git, making me think this may be related to something in our setup.pys. Might not be a bug in poetry, but this behavior is new :-)

Workaround

Current workaround is to disable the new installer in favour of the old one. (#3086 (comment))

[mchesterkadwell]

I am experiencing the same issue since the release of 1.1.0 (I can now reproduce on 1.1.0 and 1.1.1).

My Docker image build has 5 git repos pulled in by SSH from Bitbucket. Poetry reports they are installing, but after running pip list in a container built from the image it shows one or more of them will not actually be installed, seemingly at random. I have run the build multiple times and each time get a different combination of correctly installed packages.

I get the same ModuleNotFoundError: No module named 'foo' when trying to run the application.

[ponas]

A temporary workaround is to disable the new installer with:
POETRY_EXPERIMENTAL_NEW_INSTALLER="" poetry install
or
poetry config experimental.new-installer false

[abn]

This is also causing issues with source packages. I'm installing source packages from our self-hosted git repo, and there is a clear race condition when using the new installer. I originally blamed #2102, but after some experimentation determined that the issue was unrelated to that.

My symptoms are that some of my source packages do not exist in my environment after doing the install. The happens both for system installs and venvs. Digging in, I find that with python -m site the path to the source is no longer in the python path.

Further digging shows that the paths come from site-packages/easy-install.pth, which has a list of the source-installed packages. Every time I run the install, that list changes. Some of the packages just aren't there. Which ones are change. The number that are in the list changes from time to time as well.

I suspect that there is a race condition on the installation of these: multiple separate processes/threads modify the file at the same time without taking a lock.

There's a few options I can see to fix this, though I'm sure this is a non-exhaustive list:

• Ensure .pth files are installed for all source packages (it looks like some do -- ones that themselves use poetry).
• Make sure the easy-install.pth file is locked, and that whatever is modifying it takes the lock
• Process source dependencies serially.

I worked around this problem by disabling the experimental installer. Downgrading to 1.0.10 also did the trick. Neither of these are fantastic solutions. I suppose I could update all the dependencies to use poetry, but that may not always be possible, and it adds quite a bit of work in the short term to unblock my builds.

Originally posted by @klarose in #2658 (comment)

[abn]

This was also discussed on the discord server. Effectively, the issue is a race condition, when packages (here, vcs packages) are installed. These are by default installed with develop = true, and in this mode these packages end up executing python setup.py develop simultaneously in some cases. And since both these operations execute writes to easy-install.pth we end up non-deterministic behavior.

Potential bug fix might be to;

1. make these installation sequential
2. use python setup.py egg_info and get poetry to handle setuptools compatible "linking" instead of python setup.py develop.

[PetterS]

Since there seems to be several issues with parallel install/uninstall in 1.1, perhaps it would make sense to disable parallel execution by default until it is fixed in 1.2?

[abn]

@PetterS you can disable the new installer easily at the moment. And the only real issue at the moment is the develop install at the moment. I'd much rather resolve or mitigate that, than simply disable parallel all together.

[PetterS]

It isn't really documented but for anyone finding this issue and wondering how to do it, here is a command:

poetry config experimental.new-installer false --local

Several people are having trouble with parallel installation. I would not say that the "only real issue" is the develop install since the uninstalls crashes as well.

Sure, of course it would be nice to mitigate or fix, but 1.1 is a stable version which people are using for real things. Stability should come before features for a tool like this IMHO.

But you are right, disabling experimental.new-installer should hopefully fix most things.

EDIT: This was also posted higher up in the thread, which I did not see.

[abn]

@ponas @mchesterkadwell can either of you verify that the fix in #3099 resolves the issue?

[ponas]

@abn Unfortunately, this did not resolve the issue. I tried verifying by installing the same test project as before using your branch:

$ pipx install --suffix @issue3086 git+https://github.com/abn/poetry.git@issue/3086
$ poetry@issue3086 install
$ poetry@issue3086 run python -c 'import cerebrum_client'
$ poetry@issue3086 run python -c 'import tofh'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'tofh'

[abn]

@ponas was this with a clean venv?

EDIT: I think I know what the issue is, let me verify.

[abn]

@ponas can you try again please?

[ponas]

@abn Your latest commit solves my issue. Thanks for looking into this! :-)

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.