Mypycropip? #24

bollwyvl · 2022-11-14T13:25:23Z

Elevator pitch

Use mypyc during the build of micropip, offer micropip.*emscripten.*.whl on PyPI.

Motivation

It's theoretically a very small packaging change to add mypyc.

This would convert the work and extra bytes (and sometimes surprisingly non-trivial runtime overhead) of type hints into compiled C code, add more robust (and efficient) runtime type checks on functions.

Design considerations

would benefit from prior benchmarking of:
- .whl size
- time to load in interpreter
- time to first package install
- time to handle a "large" installation request e.g. 100 packages
would still want to offer a none-any.whl to keep the lightweight posture
downstreams such as jupyterlite's piplite might be left out
- but if Enhancement ideas from JupyterLite #16 ever was closed, it wouldn't need to do any of that stuff, as there would be documented, typed APIs
errors in compiled code become rather opaque, so things like Offer redirection of micropip.install console messages and data #23 would be more important

The text was updated successfully, but these errors were encountered:

rth · 2022-11-14T16:17:55Z

Thanks for the proposal! There is a somewhat related ongoing work to py-compile (convert .py files to .pyc) in pyodide/pyodide#3253 and pyodide/pyodide#3166 which should also result in significantly faster loading of the interpreter and micropip.

About mypyc I'm not sure. mypyc makes sense for functions that are compute bound, but it feels to me we are doing very little computation in micropip: most of it is I/O bound currently I would say (either via the network for the in memory FS), but maybe I'm wrong. If there are compute bottlenecks there, it might be worthwhile optimizing them first in pure Python. Some of the loading overhead was due to .py -> .pyc compilation and that would be removed when shipping .pyc.

Though yes, if we go forward with shipping .pyc files (and lose source readability anyway), why not go further and turn the source through some actual compilers (mypyc or Cython) is indeed an interesting question.

bollwyvl · 2022-11-14T21:21:19Z

Welp, it all starts with benchmarking and profiling: there are a solid number of loops where strings are compared, which seems to be a place where mypyc can do some good, but can't know until we look. I've had... poor success with getting actionable profiling out of jupyterlite's pyolite... perhaps getting pyinstrument built for pyodide, which supports profiling async methods would be enough to start slugging on it for real.

lose source readability anyway

i think once most folk see issues inside the ur-plumbing (package managers, stdlib), they kinda zone out anyway. But definitely a tradeoff... again it comes down to all those competing dimensions of file size, browser/js parsing, interpreter overhead, etc. of .py, .pyc, .py, .so, and everything else that might be in one of these packages... and maybe would lead to micropip equivalents of --prefer-binary for folk that really want to get into the nitty-gritty without hex editing.

rth · 2022-11-14T21:44:20Z

Yes, good idea! pyinstaller actually indeed works fine,

from pyinstrument import Profiler
import pyodide_js
p = Profiler()
with p:
    await pyodide_js.loadPackage('micropip')
    import micropip
p.print()

outputs,

  _     ._   __/__   _ _  _  _ _/_   Recorded: 22:33:47  Samples:  140
 /_//_/// /_\ / //_// / //_'/ //     Duration: 0.156     CPU time: 0.000
/   _/                      v4.4.0
Program: 
0.153 <module>  <console>:1
`- 0.152 <module>  micropip/__init__.py:1
   |- 0.150 <module>  micropip/_micropip.py:1
   |  |- 0.103 <module>  packaging/markers.py:1
   |  |  |- 0.067 <module>  pyparsing/__init__.py:1
   |  |  |  |- 0.023 <module>  pyparsing/core.py:1
   |  |  |  |  |- 0.007 <module>  pyparsing/results.py:1
   |  |  |  |  |  |- 0.005 compile  None
   |  |  |  |  |  |     [2 frames hidden]  <built-in>
   |  |  |  |  |  `- 0.002 str.join  None
   |  |  |  |  |        [2 frames hidden]  <built-in>
   |  |  |  |  |- 0.004 srange  pyparsing/core.py:5684
   |  |  |  |  |  `- 0.004 And.parse_string  pyparsing/core.py:1076
   |  |  |  |  |     |- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |  |  |     |  `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |     `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |  |  |     |        `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |           `- 0.002 Group.streamline  pyparsing/core.py:4409
   |  |  |  |  |     |              `- 0.002 OneOrMore.streamline  pyparsing/core.py:4409
   |  |  |  |  |     |                 `- 0.002 MatchFirst.streamline  pyparsing/core.py:4092
   |  |  |  |  |     |                    `- 0.002 MatchFirst.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |                       `- 0.002 Group.streamline  pyparsing/core.py:4409
   |  |  |  |  |     |                          `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |  |  |     |                             `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |                                `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |  |  |     |                                   `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |                                      `- 0.002 MatchFirst.streamline  pyparsing/core.py:4092
   |  |  |  |  |     |                                         `- 0.002 MatchFirst.streamline  pyparsing/core.py:3675
   |  |  |  |  |     |                                            `- 0.002 MatchFirst.streamline  pyparsing/core.py:4092
   |  |  |  |  |     |                                               `- 0.002 <genexpr>  pyparsing/core.py:4100
   |  |  |  |  |     `- 0.002 And._parseNoCache  pyparsing/core.py:776
   |  |  |  |  |        `- 0.002 And.parseImpl  pyparsing/core.py:3861
   |  |  |  |  |           `- 0.002 Group._parseNoCache  pyparsing/core.py:776
   |  |  |  |  |              `- 0.002 Group.parseImpl  pyparsing/core.py:4373
   |  |  |  |  |                 `- 0.002 OneOrMore._parseNoCache  pyparsing/core.py:776
   |  |  |  |  |                    `- 0.002 OneOrMore.parseImpl  pyparsing/core.py:4779
   |  |  |  |  |                       `- 0.002 MatchFirst._parseNoCache  pyparsing/core.py:776
   |  |  |  |  |                          `- 0.002 MatchFirst.parseImpl  pyparsing/core.py:4108
   |  |  |  |  |                             `- 0.002 Group._parseNoCache  pyparsing/core.py:776
   |  |  |  |  |- 0.003 ParserElement  pyparsing/core.py:400
   |  |  |  |  |  `- 0.002 inner  typing.py:305
   |  |  |  |  |        [4 frames hidden]  typing
   |  |  |  |  `- 0.002 <listcomp>  pyparsing/core.py:5798
   |  |  |  |     `- 0.002 ParserElement.__instancecheck__  abc.py:117
   |  |  |  |           [9 frames hidden]  abc, <built-in>
   |  |  |  |- 0.022 <module>  pyparsing/helpers.py:1
   |  |  |  |  |- 0.012 <module>  html/__init__.py:1
   |  |  |  |  |     [15 frames hidden]  html, <built-in>, re, sre_compile
   |  |  |  |  |- 0.003 make_html_tags  pyparsing/helpers.py:648
   |  |  |  |  |  `- 0.003 _makeTags  pyparsing/helpers.py:590
   |  |  |  |  |     `- 0.003 Word.__init__  pyparsing/core.py:2678
   |  |  |  |  |        `- 0.002 Word.name  pyparsing/core.py:1857
   |  |  |  |  |           `- 0.002 Word.default_name  pyparsing/core.py:1832
   |  |  |  |  |              `- 0.002 Word._generateDefaultName  pyparsing/core.py:2791
   |  |  |  |  |                 `- 0.002 charsAsStr  pyparsing/core.py:2792
   |  |  |  |  |                    `- 0.002 _collapse_string_to_ranges  pyparsing/util.py:182
   |  |  |  |  `- 0.002 <dictcomp>  pyparsing/helpers.py:692
   |  |  |  |- 0.010 <module>  pyparsing/common.py:1
   |  |  |  |  `- 0.010 pyparsing_common  pyparsing/common.py:8
   |  |  |  |     |- 0.004 Combine.__init__  pyparsing/core.py:5392
   |  |  |  |     |  `- 0.004 Combine.leave_whitespace  pyparsing/core.py:4379
   |  |  |  |     |     `- 0.004 MatchFirst.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     |        `- 0.004 And.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     |           `- 0.003 And.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     |              `- 0.002 Opt.leave_whitespace  pyparsing/core.py:4379
   |  |  |  |     |                 `- 0.002 And.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     |                    `- 0.002 Opt.leave_whitespace  pyparsing/core.py:4379
   |  |  |  |     |                       `- 0.002 And.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     |                          `- 0.002 Opt.leave_whitespace  pyparsing/core.py:4379
   |  |  |  |     |                             `- 0.002 And.leave_whitespace  pyparsing/core.py:3635
   |  |  |  |     `- 0.003 Word.__init__  pyparsing/core.py:2678
   |  |  |  |        `- 0.002 _collapse_string_to_ranges  pyparsing/util.py:182
   |  |  |  |- 0.006 <module>  pyparsing/exceptions.py:1
   |  |  |  |  |- 0.002 _lazyclassproperty.__get__  pyparsing/unicode.py:14
   |  |  |  |  |  `- 0.002 ExceptionWordUnicode.alphanums  pyparsing/unicode.py:80
   |  |  |  |  |     `- 0.002 _lazyclassproperty.__get__  pyparsing/unicode.py:14
   |  |  |  |  |        `- 0.002 ExceptionWordUnicode.alphas  pyparsing/unicode.py:70
   |  |  |  |  `- 0.002 _collapse_string_to_ranges  pyparsing/util.py:182
   |  |  |  |     `- 0.002 is_consecutive  pyparsing/util.py:185
   |  |  |  `- 0.004 [self]  None
   |  |  |- 0.031 <module>  packaging/specifiers.py:1
   |  |  |  |- 0.014 <module>  packaging/utils.py:1
   |  |  |  |  |- 0.008 <module>  packaging/version.py:1
   |  |  |  |  |  |- 0.006 Version  packaging/version.py:257
   |  |  |  |  |  |  `- 0.006 compile  re.py:249
   |  |  |  |  |  |        [26 frames hidden]  re, sre_compile, sre_parse
   |  |  |  |  |  `- 0.002 inner  typing.py:305
   |  |  |  |  |        [19 frames hidden]  typing, <built-in>
   |  |  |  |  |- 0.003 <module>  packaging/tags.py:1
   |  |  |  |  |  `- 0.002 FileIO.read  None
   |  |  |  |  |        [2 frames hidden]  <built-in>
   |  |  |  |  `- 0.002 loads  None
   |  |  |  |        [2 frames hidden]  <built-in>
   |  |  |  `- 0.014 Specifier  packaging/specifiers.py:299
   |  |  |     `- 0.014 compile  re.py:249
   |  |  |           [52 frames hidden]  re, sre_compile, sre_parse, <built-in>
   |  |  `- 0.002 inner  typing.py:305
   |  |        [6 frames hidden]  typing
   |  |- 0.025 <module>  packaging/requirements.py:1
   |  |  |- 0.022 And.parse_string  pyparsing/core.py:1076
   |  |  |  |- 0.017 And._parseNoCache  pyparsing/core.py:776
   |  |  |  |  `- 0.017 And.parseImpl  pyparsing/core.py:3861
   |  |  |  |     `- 0.017 MatchFirst._parseNoCache  pyparsing/core.py:776
   |  |  |  |        `- 0.016 MatchFirst.parseImpl  pyparsing/core.py:4108
   |  |  |  |           `- 0.016 And._parseNoCache  pyparsing/core.py:776
   |  |  |  |              `- 0.016 And.parseImpl  pyparsing/core.py:3861
   |  |  |  |                 `- 0.016 And._parseNoCache  pyparsing/core.py:776
   |  |  |  |                    `- 0.015 And.parseImpl  pyparsing/core.py:3861
   |  |  |  |                       `- 0.015 Opt._parseNoCache  pyparsing/core.py:776
   |  |  |  |                          `- 0.015 Opt.parseImpl  pyparsing/core.py:4956
   |  |  |  |                             `- 0.015 MatchFirst._parseNoCache  pyparsing/core.py:776
   |  |  |  |                                `- 0.015 MatchFirst.parseImpl  pyparsing/core.py:4108
   |  |  |  |                                   `- 0.015 Combine._parseNoCache  pyparsing/core.py:776
   |  |  |  |                                      `- 0.015 Combine.parseImpl  pyparsing/core.py:4373
   |  |  |  |                                         `- 0.015 And._parseNoCache  pyparsing/core.py:776
   |  |  |  |                                            `- 0.015 And.parseImpl  pyparsing/core.py:3861
   |  |  |  |                                               `- 0.015 Or._parseNoCache  pyparsing/core.py:776
   |  |  |  |                                                  `- 0.015 Or.parseImpl  pyparsing/core.py:3949
   |  |  |  |                                                     `- 0.015 Regex.try_parse  pyparsing/core.py:878
   |  |  |  |                                                        `- 0.015 Regex._parseNoCache  pyparsing/core.py:776
   |  |  |  |                                                           `- 0.015 Regex.parseImpl  pyparsing/core.py:2984
   |  |  |  |                                                              `- 0.015 cached_property.__get__  functools.py:961
   |  |  |  |                                                                 `- 0.015 Regex.re_match  pyparsing/core.py:2973
   |  |  |  |                                                                    `- 0.015 cached_property.__get__  functools.py:961
   |  |  |  |                                                                       `- 0.015 Regex.re  pyparsing/core.py:2961
   |  |  |  |                                                                          `- 0.015 compile  re.py:249
   |  |  |  |                                                                                [45 frames hidden]  re, sre_compile, sre_parse, <built-in>
   |  |  |  `- 0.005 And.streamline  pyparsing/core.py:3816
   |  |  |     `- 0.005 And.streamline  pyparsing/core.py:3675
   |  |  |        `- 0.005 And.streamline  pyparsing/core.py:3816
   |  |  |           `- 0.005 And.streamline  pyparsing/core.py:3675
   |  |  |              `- 0.005 And.streamline  pyparsing/core.py:3816
   |  |  |                 `- 0.005 And.streamline  pyparsing/core.py:3675
   |  |  |                    `- 0.004 MatchFirst.streamline  pyparsing/core.py:4092
   |  |  |                       `- 0.004 MatchFirst.streamline  pyparsing/core.py:3675
   |  |  |                          `- 0.004 And.streamline  pyparsing/core.py:3816
   |  |  |                             `- 0.004 And.streamline  pyparsing/core.py:3675
   |  |  |                                `- 0.003 And.streamline  pyparsing/core.py:3816
   |  |  |                                   `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |                                      `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |                                         `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |                                            `- 0.002 Opt.streamline  pyparsing/core.py:4409
   |  |  |                                               `- 0.002 MatchFirst.streamline  pyparsing/core.py:4092
   |  |  |                                                  `- 0.002 MatchFirst.streamline  pyparsing/core.py:3675
   |  |  |                                                     `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |                                                        `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |                                                           `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |                                                              `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |                                                                 `- 0.002 Combine.streamline  pyparsing/core.py:4409
   |  |  |                                                                    `- 0.002 And.streamline  pyparsing/core.py:3816
   |  |  |                                                                       `- 0.002 And.streamline  pyparsing/core.py:3675
   |  |  |                                                                          `- 0.002 Or.streamline  pyparsing/core.py:3937
   |  |  `- 0.002 Combine.__init__  pyparsing/core.py:5392
   |  |     `- 0.002 Combine.__init__  pyparsing/core.py:5368
   |  |        `- 0.002 Combine.__init__  pyparsing/core.py:4349
   |  |           `- 0.002 Combine.__init__  pyparsing/core.py:455
   |  |- 0.008 <module>  micropip/externals/pip/_internal/utils/wheel.py:1
   |  |  |- 0.005 <module>  email/parser.py:1
   |  |  |     [20 frames hidden]  email, <built-in>, re, sre_compile, s...
   |  |  `- 0.002 [self]  None
   |  |- 0.003 WheelInfo.dataclass  dataclasses.py:1157
   |  |     [9 frames hidden]  dataclasses
   |  |- 0.002 compile  None
   |  |     [2 frames hidden]  <built-in>
   |  |- 0.002 loads  None
   |  |     [2 frames hidden]  <built-in>
   |  |- 0.002 <module>  micropip/_compat.py:1
   |  |  `- 0.002 <module>  micropip/_compat_in_pyodide.py:1
   |  |- 0.002 stat  None
   |  |     [2 frames hidden]  <built-in>
   |  `- 0.002 <module>  micropip/package.py:1
   |     `- 0.002 PackageMetadata.dataclass  dataclasses.py:1157
   |           [5 frames hidden]  dataclasses
   `- 0.002 zipimporter.__init__  <frozen zipimport>:64
         [3 frames hidden]  <frozen zipimport>, <built-in>

bollwyvl · 2022-11-14T21:46:53Z

savage 🦖

pradyunsg · 2022-12-09T09:29:29Z

I believe the newer version of packaging have a lower footprint in terms of that profiling. Could you test that?

bollwyvl mentioned this issue Nov 14, 2022

Add pyinstrument 4.4.x pyodide/pyodide#3255

Closed

bollwyvl mentioned this issue Nov 15, 2022

Add CLI command to py-compile wheels pyodide/pyodide#3253

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mypycropip? #24

Mypycropip? #24

bollwyvl commented Nov 14, 2022 •

edited

Loading

rth commented Nov 14, 2022

bollwyvl commented Nov 14, 2022

rth commented Nov 14, 2022

bollwyvl commented Nov 14, 2022

pradyunsg commented Dec 9, 2022

Mypycropip? #24

Mypycropip? #24

Comments

bollwyvl commented Nov 14, 2022 • edited Loading

Elevator pitch

Motivation

Design considerations

rth commented Nov 14, 2022

bollwyvl commented Nov 14, 2022

rth commented Nov 14, 2022

bollwyvl commented Nov 14, 2022

pradyunsg commented Dec 9, 2022

bollwyvl commented Nov 14, 2022 •

edited

Loading