Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support installing data_files in micropip.install #18

Open
bollwyvl opened this issue Aug 15, 2022 · 4 comments
Open

Support installing data_files in micropip.install #18

bollwyvl opened this issue Aug 15, 2022 · 4 comments
Labels
enhancement New feature or request

Comments

@bollwyvl
Copy link

🚀 Feature

When installing .whl files that contain data_files, these should be copied to the expected locations.

Motivation

A number of package ecosystems use data_files to deploy files to non-pythonic locations. In pyodide 0.21.0, these don't appear to be honored, which leaves some packages broken, such as nbconvert.

Pitch

  • when installing a package, e.g. nbconvert-6.5.3-py3-none-any.whl
    • find data_files e.g. nbconvert-6.5.3.data/data/*
    • copy the data_files to sys.prefix e.g. /
    • see the contents, e.g. /share/jupyter/nbconvert/templates/lab/index.html.j2

Alternatives

  • choose not to support this feature, forcing workarounds

Additional context

  • here is an example of the tricks needed to overcome this
  • perhaps this would land somewhere in _package_loader.py?
@bollwyvl bollwyvl added the enhancement New feature or request label Aug 15, 2022
@rth
Copy link
Member

rth commented Aug 15, 2022

Thanks for the suggestion. +1 to add it.

It looks like the the expected behavior is documented in https://setuptools.pypa.io/en/latest/deprecated/distutils/setupscript.html#installing-additional-files

For instance, the nbconvert wheel contains,

Archive:  nbconvert-6.5.3-py3-none-any.whl
  inflating: nbconvert/__init__.py   
  inflating: nbconvert/__main__.py   
  inflating: nbconvert/_version.py   
  [...]
  inflating: nbconvert-6.5.3.data/data/share/jupyter/nbconvert/templates/asciidoc/conf.json  
  inflating: nbconvert-6.5.3.data/data/share/jupyter/nbconvert/templates/asciidoc/index.asciidoc.j2  

so somehow we need to detect the *.data folder and move that folder relative to sys.prefix (/ in the case of Pyodide). Maybe there are tools in packaging to do that.

@hoodmane
Copy link
Member

The code for this in pip is here:
https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L503

As usual for pip it looks pretty complicated...

@pradyunsg
Copy link

I would suggest looking at https://github.com/pypa/installer instead, for reference code. It might actually be reasonable for micropip to switch to that package for performing the installations too.

If you do end up going down this route, please feel welcome to mention me for inputs/review. :)

@rth
Copy link
Member

rth commented Sep 12, 2022

Thanks for the pointer @pradyunsg and I agree we should definitely try to re-use more of the existing pypa tools.

We would prefer not to add too many runtime dependencies to micropip but in this case, maybe we can indeed vendor installer a bit similar to how pip vendors packages it needs. We are already using pypa/packaging as a runtime dependency.

If you have any other feedback on micropip or the direction you think we should be going I would be very happy to hear it! The main idea is that we need a light installer, reduced functionality is OK, for instance, we don't think it makes sense to use a backtracking dependency solver when loading some web application (excluding the Jupyterlite use case which is a bit different), and most complex dependency solving should happen when deploying the app, not at runtime. The load time needs to be small and deterministic.

We have a way to freeze dependencies though it's not very well documented yet. This is probably rather related to the "lockfiles for Python" work happening elsewhere in the ecosystem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

4 participants