Projects with extension modules no longer installed in editable mode

[robbotorigami]

• Poetry version: 1.6.1
• Python version: 3.11
• OS version and name: Ubuntu 22.04
• pyproject.toml: gist

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

I've been experiencing an issue with poetry for projects that have compiled extension modules. When installing with poetry install, the package is not installed in editable mode. This does not happen with projects that do not contain extension modules, and does not occur for poetry 1.5.1. I took a look through the changelog for 1.6.0 and found this relevant comment by Teagum in PR #7975 , but I didn't see any follow up or issues that captured this.

[dimbleby]

if you intend for someone to look at this please provide a complete reproducible example. eg a toy github repository.

poetry just shells out to pip here - does pip install -e . give different results? if not, there's perhaps not much that poetry can do about it.

[robbotorigami]

Thanks for the quick response! See: https://github.com/robbotorigami/poetry-editable-extensions/tree/main

To replicate what I'm seeing:

pipx install --force poetry=={VERSION}
poetry env remove --all
poetry install
poetry run python -c 'from example import modulea; print(modulea.__file__)'

For me with VERSION=1.5.1 I get the expected/desired output of $PWD/src/example/modulea.py, however with VERSION=1.6.1 I get a location in the site packages of the venv, e.g. /home/$USER/.cache/pypoetry/virtualenvs/poetry-editable-extensions-0u8noEE1-py3.11/lib/python3.11/site-packages/example/modulea.py.

pip install -e . does not do an editable install (same as the version 1.6.1 case), but ONLY when poetry.core.masonry.api is used as the build backend. On the setuptools branch of that repo, I have an example with a setup.py based on the one generated by poetry and with the build backend changed to setuptools.build_meta. On this branch, pip install -e . does successfully install in editable mode. My steps for testing:

python3.11 -m venv venv
source venv/bin/activate
pip install -e .
python -c 'from example import modulea; print(modulea.__file__)'

On the main branch (with poetry.core.masonry.api backend), this produces a path in the venv site packages, while on the setuptools branch (with setuptools.build_meta backend), this produces the path to the source file.

While I was looking at pip's documentation, it seems that if no pyproject.toml is available, it defaults to using setuptools.build_meta:__legacy__ as the build backend, which likely explains why this behavior changed after #7975.

Based on this, I think there's some subtle difference to the behavior of the masonry backend when building an editable wheel with / without an extension module (and vs setuptools). I built a wheel using each (with pip install -e . --no-clean), and I found that while they both contain a .pth file pointing to the source directory, the wheel built with poetry also includes the full built package (inclusive of the .so and all .py files). I've attached these wheels here: built_wheels.zip

I think that this is due to the way the WheelBuilder._build function is written, it seems like it always copies the built library into the wheel regardless of if it is performing an editable install.

[dimbleby]

sounds like you're doing great - submit a merge request when you're ready!

[robbotorigami]

Now that I know where the issue is coming from I was able to identify this PR: python-poetry/poetry-core#335 , which seems to address the problem.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.