Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources #3072

Closed
thinhha opened this issue Feb 2, 2022 · 13 comments · Fixed by #3077
Closed

[BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources #3072

thinhha opened this issue Feb 2, 2022 · 13 comments · Fixed by #3077

Comments

@thinhha
Copy link

thinhha commented Feb 2, 2022

setuptools version

setuptools==60.7.0

Python version

3.9

OS

Debian 11

Additional environment information

No response

Description

FileNotFoundError when importing pkg_resources

  File "/home/aaa/env/lib/python3.9/site-packages/google/api_core/client_info.py", line 23, in <module>
    import pkg_resources
  File "/tmp/aaa/pypi__setuptools/pkg_resources/__init__.py", line 74, in <module>
    from pkg_resources.extern.jaraco.text import (
  File "/tmp/aaa/pypi__setuptools/pkg_resources/_vendor/jaraco/text/__init__.py", line 227, in <module>
    lorem_ipsum: str = files(__name__).joinpath('Lorem ipsum.txt').read_text()
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/aaa/py_deps/pypi__setuptools/pkg_resources/_vendor/jaraco/text/Lorem ipsum.txt'

Expected behavior

Pinning setuptools==60.6.0 in requirements.txt works fine. This issue started with setuptools==60.7.0.

How to Reproduce

Using Bazel and rules_python, build a zip executable using:

bazel build //<target> --output_groups=python_zip_file
python3 bazel-bin/<target>/<target>.zip 

Output

...
  File "/home/aaa/env/lib/python3.9/site-packages/google/api_core/client_info.py", line 23, in <module>
    import pkg_resources
  File "/tmp/aaa/pypi__setuptools/pkg_resources/__init__.py", line 74, in <module>
    from pkg_resources.extern.jaraco.text import (
  File "/tmp/aaa/pypi__setuptools/pkg_resources/_vendor/jaraco/text/__init__.py", line 227, in <module>
    lorem_ipsum: str = files(__name__).joinpath('Lorem ipsum.txt').read_text()
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1266, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1252, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/home/aaa/.pyenv/versions/3.9.7/lib/python3.9/pathlib.py", line 1120, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/aaa/py_deps/pypi__setuptools/pkg_resources/_vendor/jaraco/text/Lorem ipsum.txt'
@thinhha thinhha added bug Needs Triage Issues that need to be evaluated for severity and status. labels Feb 2, 2022
@thinhha thinhha changed the title [BUG] [BUG] FileNotFoundError jaraco/text/Lorem ipsum.txt when importing pkg_resources Feb 2, 2022
@abravalheri
Copy link
Contributor

abravalheri commented Feb 2, 2022

Hi @thinhha thank you very much for opening the issue.

I just double checked the wheel published on PyPI, and it seems to include the file:

image

If I build setuptools locally with python -m build, I am also able to find the file.

I also double checked in setuptools' own configuration file, and this file is being included in the package_data option:

package_data.update({'pkg_resources._vendor.jaraco.text': ['*.txt']})

which, according to this table, indicates that they should be included in the distribution.

Any chance this problem is being caused by the tool you are using to build a zip executable? (e.g. if there is a problem in bazel or rules_python preventing them for including all the files that would be present in a wheel, in the generated zip; or if this technique of bundling dependencies together does not work with importlib-resources and importlib-metadata)

Can you reproduce this issue, without using bazel?

@abravalheri abravalheri added Needs Repro Issues that need a reproducible example. Waiting User Feedback and removed bug Needs Triage Issues that need to be evaluated for severity and status. labels Feb 2, 2022
@edward-watson-oxb
Copy link

This is linked to the issue that Bazel does not play nicely with spaces in file names.

I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!

I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

@abravalheri
Copy link
Contributor

Thank you very much for the investigation @edward-watson-oxb !

Would you (or any other member of the community) be interested in providing a PR with the name change to https://github.com/jaraco/jaraco.text? (As a temporary workaround for the bazel limitation)

I think that if the reasons are well explained, the upstream project has good chances of accepting it... Otherwise I am afraid that the fix will have to come from the bazel side...

@jaraco
Copy link
Member

jaraco commented Feb 3, 2022

Actually, I'll be unlikely to accept a change to rename the file without spaces. Setuptools itself has files that have spaces in the name. See #134. Can you look into how it is that bazel is able to build setuptools<60.7 without the issue?

@jaraco
Copy link
Member

jaraco commented Feb 3, 2022

One thing that makes this issue particularly problemmatic is that the behavior is invoked on import, and it's behavior that Setuptools doesn't even need. I'm slightly tempted to move the functionality to another library entirely. It just feels like a whole lot of overhead to create an entirely new project to host functionality to be shared by pkg_resources and setuptools.

@jaraco
Copy link
Member

jaraco commented Feb 3, 2022

I may consider patching jaraco.text when it's vendored to eliminate the file or rename it.

@jaraco
Copy link
Member

jaraco commented Feb 3, 2022

I looked into making that property lazy-loaded, but that only led me to a deficiency in doctests. I'd probably have just inlined those two lines of text, except Python linters are hostile to long lines (hobgoblins of our collective minds), and I want to cleanly present the content (without introducing intervening characters to accommodate Python).

@edge0701
Copy link

edge0701 commented Feb 3, 2022

Can you look into how it is that bazel is able to build setuptools<60.7 without the issue?

As far as I can tell, the manifest file is only used on Windows. If Bazel is running on a Linux host, there is no attempt for the file to loaded.

In some cases of Bazel rules, they ignore these files entirely.

And in others, I would imagine they have to just manually patch packages when they are downloaded - but I have no public examples of this.

Edit: Or vendor those packages into their own repo and commit changes there, of course!

busunkim96 added a commit to renovate-bot/gapic-generator-python that referenced this issue Feb 7, 2022
busunkim96 added a commit to googleapis/gapic-generator-python that referenced this issue Feb 7, 2022
* chore(deps): update dependency setuptools to v60.7.1

* chore: remove link to issue

pypa/setuptools#3072 is resolved.

Co-authored-by: Anthonios Partheniou <partheniou@google.com>
Co-authored-by: Bu Sun Kim <8822365+busunkim96@users.noreply.github.com>
@rahul-theorem
Copy link

@jaraco would you be willing to reopen this? This is breaking our usage of a package that depends on cherrypy (which in turn pulls in jaraco.collections & jaraco.text) in our Bazel build, and we'd love to figure out if there's a way to move forward.

@robert-moyai
Copy link

This is linked to the issue that Bazel does not play nicely with spaces in file names.

I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!

I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

How or where did you rename the file as a hotfix @edward-watson-oxb?

@edward-watson-oxb
Copy link

This is linked to the issue that Bazel does not play nicely with spaces in file names.
I've just been bitten with this whilst using a Poetry rule from Bazel, with no way to pin the setuptools version that I can find right now. So much for my deterministic build!
I've confirmed that renaming the file to use an underscore allows Bazel to symlink it into the runfiles just fine.

How or where did you rename the file as a hotfix @edward-watson-oxb?

I can't remember exactly the path but it was within the Bazel directory after it downloaded the dependency. It wasn't really long-term solution, it was just a test to check that this was the fix required/that would solve the issue. It would be somewhere under a location like: ~/.cache/bazel/_bazel_user/[dir_checksum]/external/py_deps_setuptools/ .

If you can't fix it by bumping setuptools to a version where the filename was fixed, or pinning to a previous version, then you could probably try and apply a patch to the dependency using Bazel to make this more permanent.

@jaraco
Copy link
Member

jaraco commented Sep 24, 2024

Re-opening as #4457 re-introduced the offending behavior.

@jaraco jaraco reopened this Sep 24, 2024
@jaraco
Copy link
Member

jaraco commented Sep 24, 2024

try and apply a patch to the dependency using Bazel to make this more permanent

Indeed. By applying the workaround in #3077, it masked the flaw in Bazel and left developers there complacent to the concern. Now that Setuptools no longer maintains patched copies of dependencies, it's no longer feasible for the vendored copies to be altered to address downstream issues. Instead, the best recommendation would be for affected environments (Bazel) to patch the offending project (jaraco.text) directly (either vendored or unvendored, and preferably the latter), or even better, implement support for valid filenames.

See #4487.

@jaraco jaraco closed this as not planned Won't fix, can't repro, duplicate, stale Sep 24, 2024
@jaraco jaraco removed Needs Repro Issues that need a reproducible example. Waiting User Feedback labels Sep 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants