Add a common class to store JSON / Simple API response data

[ryanking13]

This adds an intermediate class ProjectInfo (I can't think of a better name... open to other names) that parses both JSON and Simple API PyPI responses. micropip now takes this ProjectInfo class instead of a raw PyPI JSON response when searching for packages

This is a preparation to make micropip support both JSON and Simple API.

Related: #62

[rth]

Thanks @ryanking13 that would be very useful indeed!

A couple of comments. Ping me when it's ready for a final review.

[ryanking13]

@rth Okay, I think this is ready to be reviewed.

I cached sys_tags and parse_wheel_filename since those functions were bottlenecks when I checked.

In terms of performance, from_simple_api is slower than from_json_api. This is because simple API does not have a version key in the response so we need to parse the wheel filename for every file. I added a few checks to optimize the speed but it is still around 2 times slower.

For example, this is a result when I tested with numpy, which has ~2000 files in the index:

from micropip.package_index import ProjectInfo
from pathlib import Path
from gzip import decompress
import json
import timeit

numpy_json = json.loads(decompress(Path("tests/test_data/pypi_response/numpy_json.json.gz").read_bytes()))
numpy_simple = json.loads(decompress(Path("tests/test_data/pypi_response/numpy_simple.json.gz").read_bytes()))
num_runs = 100

def run_json():
    ProjectInfo.from_json_api(numpy_json)

def run_simple():
    ProjectInfo.from_simple_api(numpy_simple)

print("Total number of files:", len(numpy_simple["files"]))
print(f"Parsing JSON API for {num_runs} times:", timeit.timeit(run_json, number=num_runs), "seconds")
print(f"Parsing Simple API for {num_runs} times:", timeit.timeit(run_simple, number=num_runs), "seconds")

Total number of files: 2185
Parsing JSON API for 100 times: 0.11372180000762455 seconds
Parsing Simple API for 100 times: 0.243828200007556 seconds

[rth]

Nice refactoring @ryanking13 ! A couple of minor comments, otherwise LGTM.

[rth]

I though we were not supporting abi3? I think there was a discussion either in the cibuildwheel or some other wheel related MR that abi3 didn't make sense for us.

If this was copied from existing code fine but maybe we should open a follow up issue.

[hoodmane]

If we are planning to break ABI with every Python version, then abi3 is a bit pointless. I think we can support it and just check that the interpreter version matches exactly. So it's supposed to be that cp311-abi3 works with all Pythons <= 311 but instead we check that the interpreter is exactly cp311. For tools like cibuildwheel that generate wheels, there's no reason to support abi3. But if someone uses a tool like maturin to generate an abi3 wheel we should be able to consume it.

[ryanking13]

Looks like we have some packages that has abi3 abi tags in Pyodide distribution.

[rth]

I cached sys_tags and parse_wheel_filename since those functions were bottlenecks when I checked

Thanks it's good to know! In what remains it looks like Version.__str__ and Version.__hash__ (e.g. when using as a dict key) is also non negligible (run with pyinstrument),

So if we want to go further one day, maybe it could be micro-optimized a bit more.. Though individually it's already quite fast,

In [1]: from packaging.version import Version

In [2]: v = Version("1.2.0")

In [3]: %timeit hash(v)
423 ns ± 2.47 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [4]: %timeit str(v)
650 ns ± 2.04 ns per loop (mean ± std. dev. of 7 runs, 1,000,000 loops each)

In [5]: %timeit str("1.2.0")
12.2 ns ± 0.0178 ns per loop (mean ± std. dev. of 7 runs, 100,000,000 loops each)

[rth]

Do we need to keep this? As long as there is a version it's fine no? And str(version) has a non negligible runtime.

[ryanking13]

Good point. I think it is okay to remove it now.

That check was introduced at pyodide/pyodide#2752, but I think it cannot happen on wheel files (unless someone modifies the version manually).

[rth]

If we want to go faster (no need to do it, just an idea). We can probably also subclass Version to something that has a frozen __str__ and __hash__ so it doesn't need to be re-computed at each dict comparison..

[ryanking13]

I opened an issue (#73) so that we can do it as a follow-up. I also think it is would be a good issue for new contributors if there is other sprint / tutorial events.

[ryanking13]

Thanks it's good to know! In what remains it looks like Version.str and Version.hash (e.g. when using as a dict key) is also non negligible (run with pyinstrument),

Right, converting string <--> Version takes quite a time... I'm not sure how many people will have performance issues with this (especially given that network latency is a much larger part of it), but it's definitely something we can optimize for. By the way, pyinstrument is cool. Thanks for introducing an awesome tool :)

[ryanking13]

Thanks for the reviews!