[BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources

[thinhha]

setuptools version

setuptools==60.7.0

Python version

3.9

OS

Debian 11

Additional environment information

No response

Description

FileNotFoundError when importing pkg_resources

  File "/home/aaa/env/lib/python3.9/site-packages/google/api_core/client_info.py", line 23, in <module>
    import pkg_resources
  File "/tmp/aaa/pypi__setuptools/pkg_resources/__init__.py", line 74, in <module>
    from pkg_resources.extern.jaraco.text import (
  File "/tmp/aaa/pypi__setuptools/pkg_resources/_vendor/jaraco/text/__init__.py", line 227, in <module>
    lorem_ipsum: str = files(__name__).joinpath('Lorem ipsum.txt').read_text()
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/aaa/py_deps/pypi__setuptools/pkg_resources/_vendor/jaraco/text/Lorem ipsum.txt'

Expected behavior

Pinning setuptools==60.6.0 in requirements.txt works fine. This issue started with setuptools==60.7.0.

How to Reproduce

Using Bazel and rules_python, build a zip executable using:

bazel build //<target> --output_groups=python_zip_file
python3 bazel-bin/<target>/<target>.zip 

Output

...
  File "/home/aaa/env/lib/python3.9/site-packages/google/api_core/client_info.py", line 23, in <module>
    import pkg_resources
  File "/tmp/aaa/pypi__setuptools/pkg_resources/__init__.py", line 74, in <module>
    from pkg_resources.extern.jaraco.text import (
  File "/tmp/aaa/pypi__setuptools/pkg_resources/_vendor/jaraco/text/__init__.py", line 227, in <module>
    lorem_ipsum: str = files(__name__).joinpath('Lorem ipsum.txt').read_text()
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/aaa/py_deps/pypi__setuptools/pkg_resources/_vendor/jaraco/text/Lorem ipsum.txt'

[abravalheri]

Hi @thinhha thank you very much for opening the issue.

I just double checked the wheel published on PyPI, and it seems to include the file:

If I build setuptools locally with python -m build, I am also able to find the file.

I also double checked in setuptools' own configuration file, and this file is being included in the package_data option:

setuptools/setup.py

Line 16 in 763cf01

package_data.update({'pkg_resources._vendor.jaraco.text': ['*.txt']})

which, according to this table, indicates that they should be included in the distribution.

Any chance this problem is being caused by the tool you are using to build a zip executable? (e.g. if there is a problem in bazel or rules_python preventing them for including all the files that would be present in a wheel, in the generated zip; or if this technique of bundling dependencies together does not work with importlib-resources and importlib-metadata)

Can you reproduce this issue, without using bazel?

[edward-watson-oxb]

This is linked to the issue that Bazel does not play nicely with spaces in file names.

I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!

I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

[abravalheri]

Thank you very much for the investigation @edward-watson-oxb !

Would you (or any other member of the community) be interested in providing a PR with the name change to https://github.com/jaraco/jaraco.text? (As a temporary workaround for the bazel limitation)

I think that if the reasons are well explained, the upstream project has good chances of accepting it... Otherwise I am afraid that the fix will have to come from the bazel side...

[jaraco]

Actually, I'll be unlikely to accept a change to rename the file without spaces. Setuptools itself has files that have spaces in the name. See #134. Can you look into how it is that bazel is able to build setuptools<60.7 without the issue?

[jaraco]

One thing that makes this issue particularly problemmatic is that the behavior is invoked on import, and it's behavior that Setuptools doesn't even need. I'm slightly tempted to move the functionality to another library entirely. It just feels like a whole lot of overhead to create an entirely new project to host functionality to be shared by pkg_resources and setuptools.

[jaraco]

I may consider patching jaraco.text when it's vendored to eliminate the file or rename it.

[jaraco]

I looked into making that property lazy-loaded, but that only led me to a deficiency in doctests. I'd probably have just inlined those two lines of text, except Python linters are hostile to long lines (hobgoblins of our collective minds), and I want to cleanly present the content (without introducing intervening characters to accommodate Python).

[edge0701]

Can you look into how it is that bazel is able to build setuptools<60.7 without the issue?

As far as I can tell, the manifest file is only used on Windows. If Bazel is running on a Linux host, there is no attempt for the file to loaded.

In some cases of Bazel rules, they ignore these files entirely.

And in others, I would imagine they have to just manually patch packages when they are downloaded - but I have no public examples of this.

Edit: Or vendor those packages into their own repo and commit changes there, of course!

[rahul-theorem]

@jaraco would you be willing to reopen this? This is breaking our usage of a package that depends on cherrypy (which in turn pulls in jaraco.collections & jaraco.text) in our Bazel build, and we'd love to figure out if there's a way to move forward.

[robert-moyai]

This is linked to the issue that Bazel does not play nicely with spaces in file names.

I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!

I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

How or where did you rename the file as a hotfix @edward-watson-oxb?

[edward-watson-oxb]

This is linked to the issue that Bazel does not play nicely with spaces in file names.
I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!
I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

How or where did you rename the file as a hotfix @edward-watson-oxb?

I can't remember exactly the path but it was within the Bazel directory after it downloaded the dependency. It wasn't really long-term solution, it was just a test to check that this was the fix required/that would solve the issue. It would be somewhere under a location like: ~/.cache/bazel/_bazel_user/[dir_checksum]/external/py_deps_setuptools/ .

If you can't fix it by bumping setuptools to a version where the filename was fixed, or pinning to a previous version, then you could probably try and apply a patch to the dependency using Bazel to make this more permanent.

[jaraco]

Re-opening as #4457 re-introduced the offending behavior.

[jaraco]

try and apply a patch to the dependency using Bazel to make this more permanent

Indeed. By applying the workaround in #3077, it masked the flaw in Bazel and left developers there complacent to the concern. Now that Setuptools no longer maintains patched copies of dependencies, it's no longer feasible for the vendored copies to be altered to address downstream issues. Instead, the best recommendation would be for affected environments (Bazel) to patch the offending project (jaraco.text) directly (either vendored or unvendored, and preferably the latter), or even better, implement support for valid filenames.

See #4487.