Support PyPy

[charliermarsh]

Right now, uv does not officially support PyPy. It might work, and we've done a few things to enable it, like:

• Add support for PyPy wheels #1028
• Use the right marker for the implementation field of pyvenv.cfg #1785
• virtualenv: determine 'site-packages' based on implementation name #2094

But we're likely missing a bunch of stuff, like:

• virtualenvs should base site-packages and other directories on sysconfig paths #2095
• uv venv doesn't add PyPy executables #2092
• Doesn't add package files when installing from sdist on PyPy #2490

[charliermarsh]

Not prioritizing this immediately, but it should happen, and I'd welcome contributors and PRs around it.

[wimglenn]

Might be a helpful data-point here - uv seems to select a wheel which it shouldn't when running with PyPy.

$ uv --version 
uv 0.1.21 (e9c12c52f 2024-03-14)
$ python3 -VV
Python 3.10.13 (fc59e61cfbff, Jan 15 2024, 14:31:34)
[PyPy 7.3.15 with GCC Apple LLVM 15.0.0 (clang-1500.1.0.2.5)]

Example reproducer:

$ cat reqs.in 
psycopg2-binary; implementation_name == 'cpython'
psycopg2; implementation_name != 'cpython'
$ uv pip compile reqs.in -v
DEBUG Starting interpreter discovery for active Python
DEBUG Found a virtualenv through VIRTUAL_ENV at: /private/tmp/p/.venv
DEBUG Probing interpreter info for: /private/tmp/p/.venv/bin/python
DEBUG Found Python 3.10.13 for: /private/tmp/p/.venv/bin/python
DEBUG Using Python 3.10.13 interpreter at /private/tmp/p/.venv/bin/python for builds
DEBUG Using registry request timeout of 300s
DEBUG Solving with target Python version 3.10.13
DEBUG Adding direct dependency: psycopg2*
DEBUG No cache entry for: https://pypi.org/simple/psycopg2/
DEBUG No credentials found for: https://pypi.org/simple/psycopg2/
DEBUG Searching for a compatible version of psycopg2 (*)
DEBUG Selecting: psycopg2==2.9.9 (psycopg2-2.9.9-cp310-cp310-win32.whl)
DEBUG No cache entry for: https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl.metadata
DEBUG No credentials found for: https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl.metadata
Resolved 1 package in 370ms
# This file was autogenerated by uv via the following command:
#    uv pip compile reqs.in
psycopg2==2.9.9

This wheel is unsupported (CPython, windows) and pip (pypy, macOS) would not consider it:

$ pip install https://files.pythonhosted.org/packages/a2/14/2767d963915f957c07f5d4c3d9c5c9a407415289f5cde90b82cb3e8c2a12/psycopg2-2.9.9-cp310-cp310-win32.whl
ERROR: psycopg2-2.9.9-cp310-cp310-win32.whl is not a supported wheel on this platform.

[charliermarsh]

Can you just confirm that the Python in your virtualenv there is PyPy too?

[wimglenn]

Yes, it is:

$ ls -l .venv/bin 
total 48248
-rw-r--r--  1 wim  wheel      9033 Mar 17 20:25 Activate.ps1
-rw-r--r--  1 wim  wheel      1989 Mar 17 20:25 activate
-rw-r--r--  1 wim  wheel       915 Mar 17 20:25 activate.csh
-rw-r--r--  1 wim  wheel      2195 Mar 17 20:25 activate.fish
-rwxr-xr-x  1 wim  wheel       235 Mar 17 20:33 pip
-rwxr-xr-x  1 wim  wheel       235 Mar 17 20:33 pip3
-rwxr-xr-x  1 wim  wheel       235 Mar 17 20:33 pip3.10
lrwxr-xr-x  1 wim  wheel         5 Mar 17 20:25 pypy -> pypy3
lrwxr-xr-x  1 wim  wheel        20 Mar 17 20:25 pypy3 -> /usr/local/bin/pypy3
lrwxr-xr-x  1 wim  wheel         5 Mar 17 20:25 python -> pypy3
lrwxr-xr-x  1 wim  wheel         5 Mar 17 20:25 python3 -> pypy3
lrwxr-xr-x  1 wim  wheel         5 Mar 17 20:25 python3.10 -> pypy3
-rwxr-xr-x  1 wim  wheel  24665448 Mar 17 20:26 uv

I created it with /usr/local/bin/pypy3 -m venv .venv and activated it, then installed uv within env.

[charliermarsh]

Thanks!

[charliermarsh]

Oh sorry, to be clear, we let the resolver use incompatible wheels during resolution for the purpose of fetching metadata. So this is fine and intended. The only problem would be if we then installed that wheel with pip sync or pip install.

[wimglenn]

Hmm, is that really fine? As far as I'm aware, there's no promise that the dependencies specified in the wheel metadata are correct for other environments. It should possible, at least in theory, for mypkg-1.2.3-cp310-cp310-win32.whl to have different dependencies than a pp310-none-any tagged wheel, or even a mypkg-1.2.3.tar.gz.

[charliermarsh]

It's an assumption we make in dependency resolution. Poetry and other tools make the same assumption. It's not standardized or guaranteed. There's discussion around it here: https://discuss.python.org/t/lock-files-again-but-this-time-w-sdists/46593/274. There's also some data on it here: https://discuss.python.org/t/insights-into-how-poetry-lock-works-cross-platform/17846/23. At least on PyPI, there are about 700 (out of 523,405) that ship with wheels with different metadata.

[ofek]

Would it be sufficiently complex to assume that each wheel has different metadata?

[charliermarsh]

Not really. It's just an optimization.

[wimglenn]

Those discuss links are very informative, thanks.

[charliermarsh]

(If anyone's looking to continue discussing the wheel metadata usage, I humbly request that we do so in a new issue.)

[olliemath]

I noticed that this seems to work pretty well now (I recently installed a large project with >150 requirements under pypy) - what's left to do on this front? Or is it mostly about battle testing various edge cases (e.g. the metadata mismatch)?

[charliermarsh]

We might be missing some edge cases around the exact virtualenv executables that are created on various platforms, but I think this is good enough that we can close it, and follow-up on any remaining issues separately as they come up.

[charliermarsh]

(I consider PyPy supported.)