Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pyodide pack: Add support for packages installed with micropip #31

Merged
merged 2 commits into from
Sep 27, 2023

Conversation

rth
Copy link
Member

@rth rth commented Sep 26, 2023

Closes #4

The added logic is,

  1. Try installing dependencies with loadPackage. If it fails, fallback to micropip
  2. If micropip is installed, at the end of the application run, call micropip.freeze() and use the output as the new lockfile for finding the install URL of wheels downloaded from PyPI
  3. Download PyPI wheels to node_modules/pyodide alongside wheels downloaded by loadPackage. This is maybe a non-optimal side effect of running pyodide pack but we should do PyPI wheel caching in node in any case.

This means however that micropip + packaging get budled unnecessarily. For now, given their relatively small size I will keep them.

Example of running pyodide pack examples/micropip_deps/app.py

┏━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ No ┃ Package                        ┃ All files ┃ .so libs ┃   Size (MB) ┃ Reduction ┃
┡━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│  0 │ stdlib                         │ 547 → 161 │          │ 2.25 → 0.82 │    63.6 % │
│  1 │ micropip-0.4.0-py3-none-any.w… │   33 → 28 │    0 → 0 │ 0.05 → 0.04 │    17.7 % │
│  2 │ packaging-23.1-py3-none-any.w… │   21 → 14 │    0 → 0 │ 0.05 → 0.04 │    24.1 % │
│  3 │ snowballstemmer-2.2.0-py2.py3… │   37 → 34 │    0 → 0 │ 0.08 → 0.08 │     3.2 % │
└────┴────────────────────────────────┴───────────┴──────────┴─────────────┴───────────┘
Wrote pyodide-package-bundle.zip with 0.17 MB (2.2% reduction)
[...]
Total output size (stdlib + packages): 0.99 MB (59.2% reduction)

@rth rth merged commit e6ced18 into pyodide:main Sep 27, 2023
4 checks passed
@rth rth deleted the micropip-support branch September 27, 2023 08:44
@ryanking13
Copy link
Member

Thanks! I guess we need some more strict tests for micropip.freeze() as we now use it here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support packages installed with micropip
2 participants