Poetry-core picks up unrelated .gitignore which results in an empty wheel

[plattfot]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.

Closest issue I found to this is #4800.

• OS version and name: CentOS 7 (not relevant to this issue)
• Poetry version: Poetry-core-1.0.8 and 1.1.0a7

Issue

I'm trying to package up poetry-core-1.0.8 at work but when building it I end up with an empty install (apart for some metadata).

private/dist/
└── python_poetry_core-1.0.8
    ├── lib
    │   └── python3.7
    │       └── site-packages
    │           └── poetry_core-1.0.8.dist-info
    │               ├── LICENSE
    │               ├── METADATA
    │               ├── RECORD
    └──             └── WHEEL

At work we are using a proprietary package manager, closest open source equivalent would probably be rez.

Each recipe to build a package is its own git repo, which we call a port. The source of a package is extracted to a subdirectory called private/build then the resulting artifact is installed to another subdirectory private/dist which then can be installed to its final destination. To keep the repo clean we list private in .gitignore.

I did some code spelunking in the source code for poetry-core and discovered that the issue is how it uses the VCS's ignore list (.gitignore for git) to pre-seed the exclude field. Since I'm downloading the release tarball, the source code does not contain any VCS info. But given that the tarball is extracted in a subdirectory (private/build) of the port which is a git repo, poetry-core picks up the .gitignore from that. And as stated earlier that ignores the whole source directory.

I patched the get_vcs(…) function to check that the path returned by git is located inside the directory given to get_vcs. And that seems to fix my issue:

diff -urB a/poetry/core/vcs/__init__.py b/poetry/core/vcs/__init__.py
--- a/poetry/core/vcs/__init__.py	2022-02-27 20:15:29.000000000 -0800
+++ b/poetry/core/vcs/__init__.py	2022-05-05 10:26:37.248645073 -0700
@@ -20,7 +20,7 @@
             )
         ).strip()
 
-        vcs = Git(Path(git_dir))
+        vcs = Git(Path(git_dir)) if git_dir.startswith(str(directory.resolve())) else None
 
     except (subprocess.CalledProcessError, OSError, RuntimeError):
         vcs = None

But I have very little knowledge of the code so this might not be the desired fix.

Repro

Using guix (with commit f49b1a10f3d13ee9597b8ef230828606f7cac99f if you want to reproduce the exact environment)
Here follows the commands to reproduce my issue. Note that guix isn't necessary if you have all the dependencies needed to build poetry-core

guix shell -D python-poetry-core python-pypa-build unzip git wget --pure -E LANG
mkdir /tmp/poetry-core-port
cd /tmp/poetry-core-port
git init
echo private > .gitignore
wget https://github.com/python-poetry/poetry-core/archive/refs/tags/1.0.8.tar.gz
mkdir -p private/build
tar -xf /tmp/poetry-core-port/1.0.8.tar.gz -C private/build
cd private/build/poetry-core-1.0.8
python -m build -wn
unzip -l dist/poetry_core-1.0.8-py2.py3-none-any.whl

Or as a oneliner:

guix shell -D python-poetry-core python-pypa-build unzip git wget --pure -- sh -c "mkdir /tmp/poetry-core-port && cd /tmp/poetry-core-port && git init && echo private > .gitignore && wget https://github.com/python-poetry/poetry-core/archive/refs/tags/1.0.8.tar.gz && mkdir -p private/build && tar -xf /tmp/poetry-core-port/1.0.8.tar.gz -C private/build && cd private/build/poetry-core-1.0.8 && python -m build -wn && unzip -l dist/poetry_core-1.0.8-py2.py3-none-any.whl"

The last output will be:

Archive:  dist/poetry_core-1.0.8-py2.py3-none-any.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
     1062  01-01-1980 00:00   poetry_core-1.0.8.dist-info/LICENSE
       87  01-01-2016 00:00   poetry_core-1.0.8.dist-info/WHEEL
     4107  01-01-2016 00:00   poetry_core-1.0.8.dist-info/METADATA
      310  01-01-2016 00:00   poetry_core-1.0.8.dist-info/RECORD
---------                     -------
     5566                     4 files

If you remove git from the environment it will successfully package up all files.

I have also tested this with poetry-core-1.1.0a7 and that exhibits the same issue.

[autinerd]

I spent a whole day debugging why my AUR package is empty when packaging a Python project which uses Poetry. AUR packages are git repos and they often contain a .gitignore to exclude the sources und build environments. But because Poetry for some reason uses that .gitignore, an empty package is built.

[mkoeppe]

In https://trac.sagemath.org/ticket/33866 we have adopted the above patch to poetry/core/vcs/__init__.py as well

[adamjstewart]

I'm a developer for the Spack package manager. Spack builds all software from source, including packages using the poetry build system. We've gotten several reports of users who have ended up with empty or broken poetry installs due to this bug:

• Installation issue: py-pkgconfig needs py-poetry, not py-poetry-core? spack/spack#29308 @climbfuji
• Installation issue: py-poetry-core spack/spack#32375 @skosukhin
• new packages (py-metaphlan, py-hclust2, iqtree2) + updates to others (py-dendropy, py-phylophlan, py-pkgconfig) spack/spack#32936 (comment) @luke-dt

Is there any work in progress to resolve this issue, or should we patch poetry using the above patch?

[neersighted]

The above patch is a shotgun pointed at unknown places, but if it works for you, more power to you. We have a user who has been working on fixing this issue/gathering input on how to solve it without breaking other things, but I have yet to see them put a PR up -- however, it is intended that we fix this in Poetry.

[adamjstewart]

Glad to know that this is a WIP! One of our users suggested setting GIT_DIR in our package manager to prevent poetry from pulling in global gitignore files, maybe that could be a workaround as well.

[neersighted]

GIT_DIR is probably a more sane workaround -- to be clear, since nobody has broken down what is happening here simply:

This occurs when you have a Poetry-managed project that is not a VCS repository (most often a tarball) extracted into the tree of a git repository, and the path that it is extracted to is listed in the .gitignore of the parent repository. In that case, Poetry can recurse above the pyproject.toml when looking for a VCS repository.

Technically the search for a .gitignore is perfectly correct -- we never look higher than the repository root. The problem is that the search for a repository root can go 'past' the pyproject.toml and the root of the Poetry project.

[strue36]

The above patch is a shotgun pointed at unknown places, but if it works for you, more power to you. We have a user who has been working on fixing this issue/gathering input on how to solve it without breaking other things, but I have yet to see them put a PR up -- however, it is intended that we fix this in Poetry.

Do you know what the progress is with this PR?

[neersighted]

Do you know what the progress is with this PR?

I have not heard anything in a couple weeks; if you're interested in trying to solve this yourself, I would suggest that you go ahead.

[yurivict]

The problem is that poetry mis-interprets the .gitignore files.

The .gitignore file in the root of the FreeBSD repository has:

/*/*/work
/*/*/work-*

This is supposed to only affect x/y/work and x/y/work-xx.

Why does poetry apply these ignore lines at all? The .gitignore file that exists up in the hierarchy is irrelevant to the currently built project.

This needs to be fixed.

[jameshilliard]

I spent a whole day debugging why my AUR package is empty when packaging a Python project which uses Poetry.

Yeah, spent a whole day tracking this down myself when packaging poetry-core for buildroot, hopefully python-poetry/poetry-core#611 should fix it properly.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.