bdist_wheel makes absolute data_files relative to site-packages #92
Comments
|
Original comment by Marcus Smith (Bitbucket: qwcode, GitHub: qwcode): I.e. it's either that the wheel spec has to grow to cover absolute data_files (I don't see how it could handle them now; putting them into {distribution}-{version}.data doesn't help because that's relative to sys.prefix), or bdist_wheel just needs to fail to build in that case. |
|
Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth): Absolute paths need to be allowed but it may be acceptable to restrict to absolute paths within the sdist. There's a place in setuptools where certain kinds of paths cause errors and I run into it from time to time. I don't remember the details atm, only that it would be much easier to use if it did allow absolute paths. |
|
Original comment by Marcus Smith (Bitbucket: qwcode, GitHub: qwcode): Why does it have to be allowed? If bdist_wheel and sdist were consistent, that would be one thing, but they're not and can't be at the current time, so it seems wrong for wheels to build absolute paths and then place them into site-packages |
|
Original comment by Donald Stufft (Bitbucket: dstufft, GitHub: dstufft): I don't see any reason why absolute paths have to be allowed. I think they are a bad design in general, everything should be rooted in sys.prefix. It's not a very good thing for a Wheel to be able to override /etc/hosts for instance. |
|
Original comment by Marcus Smith (Bitbucket: qwcode, GitHub: qwcode): btw, there's a metadata issue open for whether wheel would grow the ability to handle platform-specific paths (including absolute I guess) https://bitbucket.org/pypa/pypi-metadata-formats/issue/13/add-a-new-subdirectory-to-allow-wheels-to for me, this issue isn't about that discussion. it's about the oddity of placing absolute paths into site-packages since wheel has no ability to properly place absolute files currently, it shouldn't build projects that declare them |
|
Original comment by Daniel Holth (Bitbucket: dholth, GitHub: dholth): packagename-1.0.data/data/ is currently a way to place absolute files. This is an accidental feature but I don't have any particular beef with it. They are absolute relative to the root of the virtualenv :-) Or if no virtualenv is in use, probably / |
|
Original comment by Marcus Smith (Bitbucket: qwcode, GitHub: qwcode): so take this setup.py which defines an absolute data files at "/opt/data_file": https://gist.github.com/qwcode/9144129 build an sdist and wheel and then install each, and see where "data_file" goes.
on the other hand, relative data files get packaged into *.data/data and get installed relative to |
|
Original comment by Michael Hoglan (Bitbucket: mhoglan, GitHub: mhoglan): Graphite does a similar thing, not specifically their data files, but the lib files are specified in an absolute location (/opt/graphite/webapp) in the setup.cfg, and it results in the files being under site-packages/opt/graphite/... when you build a wheel and install it in a virtualenv. When building from source, I would specify --install-options to change those locations to be relative to the virtualenv, but that does not seem possible to pass those options into pip wheel. Removing the prefix / lib configurations in the setup.cfg cause the wheel and source installs to behave the same (ends up in site-packages); Altering the wheel and getting rid of the /opt/graphite/webapp at the top level achieves the same thing (since it would have assumed prefix of . at install); btw, I would say its not good practice for a module to be specifying absolute paths... not virtualenv friendly, and I would hope projects would fix that. I see this as more of having to work with projects that are not defined cleanly. And probably allowing there to be consistency between a src install and a wheel install. |
|
Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming):
Agreed!
Well, actually the current problem is to work with package installers and virtualenvs that are defined cleanly! Problem is that you may be able to put a data file somewhere using That's the main problem I'm facing with setuptools right now... when using setuptools, all paths for the |
|
Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne): For reference, this is bug is essentially the same as #120 |
|
Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne): @jck2 the simplest way for me is to only use package data effectively stored in a package directory side by side with the python code that needs them and never use data files. As a simple example of this approach:
|
|
Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming): My work-around for pip 7.0 (because pip automatically creates wheels from sdists) is to include this in setup.py: Pip will automatically skip the .whl packaging and run the normal sdist installation. Why on earth this decision to make an unfinished packaging system deploy things that weren't intended for it by default is beyond my belief :( People who've made |
|
Original comment by Benjamin Reedlunn (Bitbucket: breedlun, GitHub: breedlun): I just released my first python package, and it is affected by this issue. I would like to avoid absolute paths, as suggested, but I do not know the proper way. Can someone give me a hand? Here is a link to my stack overflow question that goes into more detail. |
|
Original comment by joe_code (Bitbucket: joe_code, GitHub: Unknown): This is a real problem for me as well. My setup.py script works as expected with regard to data_files that use an absolute path and honors them when I do 'python setup.py install' however when I do 'python setup.py bdist_wheel' and then pip install my wheel the data_files that I specified with an absolute path and were correctly installed using a straight setup.py install ARE NOT installed correctly from the wheel and wind up relative to site-packages. I.e. site-packages/usr/lib/blah/blah If I want to install a file outside of site-packages (say to an arbitrary place on the filesystem) I should be able to do that. The behaviour is inconsistent. I'd really like to see this fixed because right now I can't use wheels and that's exactly what I want to use. |
|
Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming): @joe_code - I can recommend finding a workaround, not using setup.py's |
|
Original comment by joe_code (Bitbucket: joe_code, GitHub: Unknown): Hey Benjamin, thanks for your reply. Could you elaborate a little bit on your solution please? |
|
Original comment by Benjamin Bach (Bitbucket: benjaoming, GitHub: benjaoming): In our case, we could factor out most of the files in In case you don't want to create OS installers, you can have a "run first" approach for your application for which you do |
|
Original comment by Erik Bray (Bitbucket: embray, GitHub: embray): I think it's a real problem that this decision has effectively broken a use case that many packages have relied in--in some cases for bad reasons, but in other cases for good reasons. Although I personally feel like the reasoning behind the breakage has some merit, breaking things without offering some kind of guidance on how best to handle outside-Python resource files has created yet another sore point against Python packaging that has been raised by some of colleagues, and it's a valid complaint. I think the argument "well we shouldn't just allow installing files to arbitrary system locations" is well meaning but ultimately spurious. It's true that, depending on what I think a better approach would be to not make arbitrary decisions for software developers who know what they're doing, and where necessary protect users (and developers who don't know what they're doing) by not allowing pip to overwrite files that already exist on their system (especially for "data files"). |
|
Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne): @embray but is pip aware of being in a virtualenv at all? |
|
Original comment by pombredanne NA (Bitbucket: pombredanne, GitHub: pombredanne): Actually it is: https://github.com/pypa/pip/blob/d86d1713647f791979b9267ffc5773479d0ef469/pip/locations.py#L39 |
|
Original comment by Benno Fünfstück (Bitbucket: bennofs, GitHub: bennofs): Is there a way right now to install some file into |
|
The man pages case or documentation/licensing is the easy one. The
application doesn't itself access them at runtime. As long as they are
categorized in the archive they can be installed anywhere or nowhere and
the program still works, but if the user cares about man data they can
put it on the path.
On Thu, Sep 13, 2018, at 2:41 PM, Nathaniel J. Smith wrote:
> 1. Must support installation in any Python environment, including
> the system Python, user-site, and a virtualenv.> I just want to note that the other 2 requirements pretty much follow
from this one. If we want to support man pages, the way to do that is
to extend the definition of a "Python environment" to include a man-
pages directory, which would require figuring out what that means in
all of these cases.> Wheels are a high-level representation of a Python package, abstracted
over the specific details of the installation environment. If you want
a high-level representation of an arbitrary application, that's just a
different thing, and wheels are not well-suited to that problem. There
are many other tools that *are* designed to solve that problem, like
rpms, debs, MacOS/Windows application installers, etc.> — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub[1], or mute the thread[2].
|
|
If you have data you want to access at runtime, then we already have a standard and well-supported solution (it even works for packages installed in zips!): https://docs.python.org/3.7/library/importlib.html#module-importlib.resources |
|
I've suggested that if a wheel contained a package-1.0.data/docs/
directory, that the installer could place those files into e.g. $virtualenv/share/docs/$packagename-
$packageversion by default. Imagine that plus a few more categories.
On Thu, Sep 13, 2018, at 3:07 PM, Nathaniel J. Smith wrote:
If you have data you want to access at runtime, then we already have a
standard and well-supported solution (it even works for packages
installed in zips!):
https://docs.python.org/3.7/library/importlib.html#module-importlib.resources> — You are receiving this because you were mentioned. Reply to this
email directly, view it on GitHub[1], or mute the thread[2].
|
Indeed. If someone wanted to flesh out that proposal, put it into the form of a PEP/standard and get it approved and then implemented in the various tools, then that would probably cover a lot of the use cases I've seen mentioned in the past. Of course, no-one has yet volunteered to champion the suggestion. It really needs someone with an actual stake in the issue to step up, or it's going to forever sit behind other priorities. |
|
Hey someone else finished my pep once. I can dream.
|
Wrong! The |
That's exactly the use case that |
You are confusing two different use cases for
|
|
what's this other software, that has nothing to do with Python, but it understands about Python environment layouts, including the data directory that even most Python software doesn't understand, but that doesn't know how to find |
"environments" are not specific to Python at all. Most open source software packages have a concept of installation prefix, analogous to |
Jupyter packages are a good example. While many Jupyter kernels are written using Python, that is not a requirement: it is possible to implement the Jupyter protocol without Python. So they decided to use |
|
And the |
|
The consensus (?) seems to be that this needs a new standard and that wheel itself is currently not doing anything wrong. If someone wants this to be reopened, be specific about what changes are required for the wheel project. Otherwise a new issue could be opened when a new standard emerges that requires implementation here. |
This change modifies the ironic base container to copy rootwarp filters from the virtual env rather than the source code directory. This is need because some required filters have been moved to ironic-lib and are not present in the /ironic dir. The rootwrap filters are not automitaclly installed in /etc/... due to kolla use of virtual envs and pypa/wheel#92 Closes-Bug: #1886663 Change-Id: Idb0a675d92bab8b9a0cf5209f0a06e996e96033c
* Update kolla from branch 'master'
- Merge "copy rootwarp files form venv in ironic base"
- copy rootwarp files form venv in ironic base
This change modifies the ironic base container
to copy rootwarp filters from the virtual
env rather than the source code directory. This
is need because some required filters have
been moved to ironic-lib and are not present in
the /ironic dir. The rootwrap filters are not
automitaclly installed in /etc/... due to kolla
use of virtual envs and pypa/wheel#92
Closes-Bug: #1886663
Change-Id: Idb0a675d92bab8b9a0cf5209f0a06e996e96033c
This change modifies the ironic base container to copy rootwarp filters from the virtual env rather than the source code directory. This is need because some required filters have been moved to ironic-lib and are not present in the /ironic dir. The rootwrap filters are not automitaclly installed in /etc/... due to kolla use of virtual envs and pypa/wheel#92 Closes-Bug: #1886663 Change-Id: Idb0a675d92bab8b9a0cf5209f0a06e996e96033c (cherry picked from commit b6c7110)
This adds the script created by Stefan in PR #57 as a starting point for further improvements on this really useful tool. Automatically installing this file with pip is something between impossible and annoying for multiple reasons. - There's no way to select between Windows and Linux, with and without bash, etc. - The file could be installed into /etc/bash_completion.d by setuptools, but not by wheels. (Not a big problem right now, since fusesoc doesn't contain binary components.) Reference: pypa/wheel#92 - Installing the file next to the package is tricky, however could be done using data_files & friends in setup.py. However that is rather tricky and provides no clear benefit over getting the file as needed with curl. So I'd go with just placing this file in the source tree and documenting its use with $ sudo curl https://raw.githubusercontent.com/olofk/fusesoc/master/extras/bash-completion -o /etc/bash_completion.d or (without root privileges) $ curl https://raw.githubusercontent.com/olofk/fusesoc/master/extras/bash-completion >> ~/.bash_completion
Originally reported by: Marcus Smith (Bitbucket: qwcode, GitHub: qwcode)
bdist_wheel doesn't handle absolute paths in the "data_files" keyword like standard setuptools installs do, which honor it as absolute (which seems to match the examples in the distutils docs)
when using absolute paths, the data ends up in the packaged wheel at the top level, and get's installed relative to site-packages (along with the project packages)
so, bdist_wheel is re-interpreting distutil's "data_files" differently. maybe better for wheel to fail to build projects with absolute data_files, than to just reinterpret it.
The text was updated successfully, but these errors were encountered: