Support installing data_files in micropip.install

[bollwyvl]

🚀 Feature

When installing .whl files that contain data_files, these should be copied to the expected locations.

Motivation

A number of package ecosystems use data_files to deploy files to non-pythonic locations. In pyodide 0.21.0, these don't appear to be honored, which leaves some packages broken, such as nbconvert.

Pitch

• when installing a package, e.g. nbconvert-6.5.3-py3-none-any.whl
  • find data_files e.g. nbconvert-6.5.3.data/data/*
  • copy the data_files to sys.prefix e.g. /
  • see the contents, e.g. /share/jupyter/nbconvert/templates/lab/index.html.j2

Alternatives

• choose not to support this feature, forcing workarounds

Additional context

• here is an example of the tricks needed to overcome this
• perhaps this would land somewhere in _package_loader.py?

[rth]

Thanks for the suggestion. +1 to add it.

It looks like the the expected behavior is documented in https://setuptools.pypa.io/en/latest/deprecated/distutils/setupscript.html#installing-additional-files

For instance, the nbconvert wheel contains,

Archive:  nbconvert-6.5.3-py3-none-any.whl
  inflating: nbconvert/__init__.py   
  inflating: nbconvert/__main__.py   
  inflating: nbconvert/_version.py   
  [...]
  inflating: nbconvert-6.5.3.data/data/share/jupyter/nbconvert/templates/asciidoc/conf.json  
  inflating: nbconvert-6.5.3.data/data/share/jupyter/nbconvert/templates/asciidoc/index.asciidoc.j2  

so somehow we need to detect the *.data folder and move that folder relative to sys.prefix (/ in the case of Pyodide). Maybe there are tools in packaging to do that.

[hoodmane]

The code for this in pip is here:
https://github.com/pypa/pip/blob/main/src/pip/_internal/operations/install/wheel.py#L503

As usual for pip it looks pretty complicated...

[pradyunsg]

I would suggest looking at https://github.com/pypa/installer instead, for reference code. It might actually be reasonable for micropip to switch to that package for performing the installations too.

If you do end up going down this route, please feel welcome to mention me for inputs/review. :)

[rth]

Thanks for the pointer @pradyunsg and I agree we should definitely try to re-use more of the existing pypa tools.

We would prefer not to add too many runtime dependencies to micropip but in this case, maybe we can indeed vendor installer a bit similar to how pip vendors packages it needs. We are already using pypa/packaging as a runtime dependency.

If you have any other feedback on micropip or the direction you think we should be going I would be very happy to hear it! The main idea is that we need a light installer, reduced functionality is OK, for instance, we don't think it makes sense to use a backtracking dependency solver when loading some web application (excluding the Jupyterlite use case which is a bit different), and most complex dependency solving should happen when deploying the app, not at runtime. The load time needs to be small and deterministic.

We have a way to freeze dependencies though it's not very well documented yet. This is probably rather related to the "lockfiles for Python" work happening elsewhere in the ecosystem.