Support PyPy 3.8 #2206

mattip · 2021-10-04T08:37:50Z

Thanks for contributing, make sure you address all the checklists (for details on how see

ran the linter to address style issues (tox -e fix_lint)
wrote descriptive pull request text
ensured there are test(s) validating the fix
added news fragment in docs/changelog folder
updated/extended the documentation

This builds on and replaces #2204. In an earlier version of portable PyPy (one built on manylinux 2010 and bundling dependent shared objects), it was necessary to copy the bundled shared objects from prefix/lib. This is no longer needed since now the libpypy3-c.so is symlinked not copied.

On non-portable PyPy installations, like those in distros or on conda-forge, this code was never used since there the dependent shared objects are provided outside the python tree.

Also fixed the stdlib which had python not pypy as the stem of the directory name.

I tested this locally to make sure it works as advertised.

gaborbernat · 2021-10-04T08:40:06Z

src/virtualenv/create/via_global_ref/builtin/pypy/pypy3.py

-    def sources(cls, interpreter):
-        for src in super(PyPy3Posix, cls).sources(interpreter):
-            yield src
-        host_lib = Path(interpreter.system_prefix) / "lib"
-        if host_lib.exists() and host_lib.is_dir():
-            for path in host_lib.iterdir():
-                yield PathRefToDest(path, dest=cls.to_lib)


I'm fairly sure there was a pypy3 variant that worked like this (over a year ago when I wrote this). Doesn't this drop support for those?

Ok, let's err on the side of caution and stick with @mgorny 's fix

I'd actually prefer if we'd not do this 😊 but make it version-specific; if you can dig out from pypy when this change happened?

Before PyPy's reorganization of the directory layout for python3.8 there were no directories in prefix/lib.

It seems this code was added in #1503 in a blanket change to "support c-extensions inside virtual environments". The issue pointed to is #1448, but that issue only mentions include files, not lib files. It seems to relate to issue #1023, but that one is windows only and deals with the libs directory, not the lib one.

Hmm, in that case I'm happy to go with your original proposal, sorry for the back and forth.

cool, updating

gaborbernat

Seems the pypy3 test suite fails 🤔

mattip · 2021-10-04T12:14:08Z

~~Hmm. When I run ../pypy3.6-HEAD/bin/pypy -m pytest tests/unit/create I get an error error: unrecognized arguments: --no-success-flaky-report. This is with pytest 6.2.5.~~ I needed to install flaky.

We should add a pypy3.7 run.

gaborbernat · 2021-10-04T12:47:24Z

We should add a pypy3.7 run.

Yes, we should have both 3.6 and 3.7 there 😊

gaborbernat

Assuming the CI passes 😊

mattip · 2021-10-04T13:30:11Z

Cool, thanks

.github/workflows/check.yml

gaborbernat

mattip · 2021-10-04T16:13:20Z

I think the linux pypy3.8 tests will be fixed with rc2 which should be ready this week. I am not sure what is the problem with windows, is it missing python.exe ?

gaborbernat · 2021-10-04T16:16:29Z

I think the linux pypy3.8 tests will be fixed with rc2 which should be ready this week. I am not sure what is the problem with windows, is it missing python.exe ?

Could be this tox-dev/filelock#101? Does pypy support https://docs.python.org/3/library/msvcrt.html#msvcrt.locking? Pypy on Windows is sometimes interesting 🤔 but don't have time to debug it...

mattip · 2021-10-04T16:30:52Z

Could be this tox-dev/filelock#101?

Maybe

Does pypy support https://docs.python.org/3/library/msvcrt.html#msvcrt.locking?

Nominally, but there may be bugs since it is implemented via CFFI, so maybe there are threading problems ...

gaborbernat · 2021-10-04T16:39:18Z

Guess this is blocked until RC2 then? Ping me once that happens.

mattip · 2021-10-04T16:52:08Z

ok, but I would like to leave windows out for now, otherwise we won't have a virtualenv story for the 3.8 release.

gaborbernat · 2021-10-04T16:59:01Z

Generally, we don't tend to do piecewise support 🤔 so if possible would prefer to fix Windows... 🤔 or at least mark the failing tests as skip explicitly 🤔

mattip · 2021-10-04T19:12:50Z

I need some help to debug the create_no_seed failures on windows. Although the source location has ['pypy.exe', `pypy3.exe`, `pypy3w.exe`, pypyw.exe`, `python.exe`, `pythonw.exe`], the last two are not copied to the destination, and the check for python.exe fails.

virtualenv/tests/unit/create/test_creator.py

Line 216 in 9cd075e

assert exe_path.exists(), "\n".join(str(i) for i in creator.bin_dir.iterdir())

.

But when I stop in a debugger where the exes are listed, python appears in the list.

virtualenv/src/virtualenv/create/via_global_ref/builtin/pypy/pypy3.py

Lines 21 to 22 in 9cd075e

    
           def exe_names(cls, interpreter): 
        
               return super(PyPy3, cls).exe_names(interpreter) | {"pypy"}

.

Why isn't it being copied?

gaborbernat · 2021-10-05T07:18:43Z

@mattip The tests failing are not the ones testing that our creator is correct, but rather than the built-in venv behaves equivalently to ours. And in this case seems pypy differs, as the python.exe is not created when using pypy builtin venv 🤔 the weird part here though is that neither 3.6 should pass... but for some reason does (only in the CI locally for me fails both). So guess the question is that should the venv module on Windows create python.exe? And if not we should patch the test to express such 😊 virtualenv does create this because people often expect python.exe to work even if one is in pypy... but seems venv from pypy does not.

mattip · 2021-10-05T08:06:01Z

the weird part here though is that neither 3.6 should pass

The 3.6 windows downloads were last updated for release 7.3.3, and have only pypy3.exe and pypy3w.exe. We started adding all the other exe variants three months ago, so only the 7.3.6rc1 downloads have pypy.exe, python.exe and python3.exe

mattip · 2021-10-05T08:35:35Z

Ahah. Found the problem with venv. In symlink_or_copy, it intercepts python.exe and replaces it with venvlauncher.exe, which does not exists on PyPy.

gaborbernat · 2021-10-05T08:38:48Z

Ahah. Found the problem with venv. In symlink_or_copy, it intercepts python.exe and replaces it with venvlauncher.exe, which does not exists on PyPy.

Is that within pypy venv module?

mattip · 2021-10-05T08:43:05Z

Yes.

mattip · 2021-10-05T08:52:46Z

cool. Fixing venv makes the failures go away locally. I will put that in for rc2 and then we can try it here in CI.

gaborbernat · 2021-10-05T09:00:37Z

👍

mattip · 2021-10-06T16:21:43Z

Something is off: bin/pypy3: error while loading shared libraries: libpypy3-c.so: cannot open shared object file: No such file or directory. :(

mattip · 2021-10-06T17:42:43Z

It seems I should pay better attention to stdlib venv test failures. The problem is there, not with virtualenv

gaborbernat · 2021-10-06T18:00:19Z

virtualenv seems like a good place to test the venv module 😃

mattip · 2021-10-12T14:20:27Z

PyPy3.8 on windows hangs waiting for a lock in a __del__. Would it make sense to refactor this code to avoid doing anything in the destructor as long as _del_lock(lock) is called with None, and then in the __exit__ set self.lock to None once it is released?

virtualenv/src/virtualenv/util/lock.py

Lines 92 to 95 in a40640c

    
           def _del_lock(lock): 
        
               with _store_lock: 
        
                   if lock is not None: 
        
                       with lock.thread_safe:

refactored to

    def _del_lock(lock):
        if lock is not None:
            with _store_lock:
                with lock.thread_safe:

  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\app_data\via_disk_folder.py", line 99, in py_info_clear
    with py_info_folder.lock_for_key(filename.stem):
  File "C:\hostedtoolcache\windows\PyPy\3.8.12\x64\Lib\contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\util\lock.py", line 132, in lock_for_key
    lock = self._create_lock(name)
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\util\lock.py", line 88, in _create_lock
    _lock_store[lock_file] = _CountedFileLock(lock_file)
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\util\lock.py", line 18, in __init__
    parent = os.path.dirname(lock_file)
  File "C:\hostedtoolcache\windows\PyPy\3.8.12\x64\Lib\ntpath.py", line 223, in dirname
    return split(p)[0]
  File "C:\hostedtoolcache\windows\PyPy\3.8.12\x64\Lib\ntpath.py", line 186, in split
    seps = _get_bothseps(p)
  File "C:\hostedtoolcache\windows\PyPy\3.8.12\x64\Lib\ntpath.py", line 38, in _get_bothseps
    return '\\/'
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\coverage\pytracer.py", line 101, in _trace
    if THIS_FILE in frame.f_code.co_filename:
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\util\lock.py", line 100, in __del__
    self._del_lock(self._lock)
  File "D:\a\virtualenv\virtualenv\.tox\pypy3\lib\site-packages\virtualenv\util\lock.py", line 93, in _del_lock
    with _store_lock:

mattip · 2021-10-12T14:32:14Z

Is coverage running for PyPy? That makes the code run without the JIT which significantly slows things down. Would it be OK to toggle coverage off for PyPy?

mattip · 2021-10-12T14:48:43Z

Additionally, macos PyPy 3.8 failed test_app_data_pinning with

ERROR: No matching distribution found for pip==19.3.1

A transient network error?

The PyPy3 logic creates symlinks for all files from the library directory existing alongside the PyPy executable. This is meant to ensure that the bundled libraries to which PyPy is linked can also be found from inside the virtualenv. However, this logic also symlinks all directories which is unnecessary and causes library directory collisions with the new install layout. Change to logic to symlink non-directories only. A similar fix has been applied to the internal venv module in PyPy3.8: https://foss.heptapod.net/pypy/pypy/-/commit/713b2af9abd2b9453e12c60143e17431a1aefb33 Fixes pypa#2182

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

mattip · 2021-10-15T11:34:00Z

Rebased off main to get #2213. CI passed except codecov.

mattip · 2021-10-18T03:21:12Z

PyPy has released a python3.8 that requires this change.

rodrigodc · 2021-10-18T18:04:14Z

I confirm that this PR fixes the problem for PyPy 3.8, released yesterday.

gaborbernat reviewed Oct 4, 2021

View reviewed changes

mattip force-pushed the pypy3-sym branch from 53372ef to 6587467 Compare October 4, 2021 09:02

gaborbernat force-pushed the pypy3-sym branch from dccb774 to 0e78a15 Compare October 4, 2021 10:10

gaborbernat requested changes Oct 4, 2021

View reviewed changes

mattip force-pushed the pypy3-sym branch 2 times, most recently from b6a1c66 to eccacfe Compare October 4, 2021 13:08

gaborbernat force-pushed the pypy3-sym branch from eccacfe to b61e4ae Compare October 4, 2021 13:21

gaborbernat approved these changes Oct 4, 2021

View reviewed changes

pypa deleted a comment from codecov bot Oct 4, 2021

mattip commented Oct 4, 2021

View reviewed changes

.github/workflows/check.yml Outdated Show resolved Hide resolved

gaborbernat approved these changes Oct 4, 2021

View reviewed changes

gaborbernat changed the title ~~Pypy3 sym~~ Support PyPy 3.8 Oct 4, 2021

mattip mentioned this pull request Oct 13, 2021

refactor to not do too much in ReentrantFileLock.__del__ #2213

Merged

5 tasks

mgorny and others added 7 commits October 15, 2021 13:48

fix stdlib path

d7ec926

skip linking/copying the stdlib specifically, extensively comment

e1f065f

Extend CI for pypy versions

3f06b32

Signed-off-by: Bernát Gábor <bgabor8@bloomberg.net>

Use pypy-3.8-v7.3.6rc1

8b217ab

try pypy v7.3.6rc2

0ac7db9

use pypy v7.3.6rc3

5381955

mattip force-pushed the pypy3-sym branch from 49b573a to 5381955 Compare October 15, 2021 10:49

rodrigodc mentioned this pull request Oct 18, 2021

Support PyPy 3.8 python-poetry/poetry#4655

Closed

3 tasks

gaborbernat merged commit 4188ac6 into pypa:main Oct 23, 2021

Support PyPy 3.8 #2206

Support PyPy 3.8 #2206

Conversation

mattip commented Oct 4, 2021 • edited Loading

Thanks for contributing, make sure you address all the checklists (for details on how see

gaborbernat Oct 4, 2021

Choose a reason for hiding this comment

mattip Oct 4, 2021

Choose a reason for hiding this comment

gaborbernat Oct 4, 2021

Choose a reason for hiding this comment

mattip Oct 4, 2021

Choose a reason for hiding this comment

gaborbernat Oct 4, 2021

Choose a reason for hiding this comment

mattip Oct 4, 2021

Choose a reason for hiding this comment

gaborbernat left a comment

Choose a reason for hiding this comment

mattip commented Oct 4, 2021 • edited Loading

gaborbernat commented Oct 4, 2021

gaborbernat left a comment

Choose a reason for hiding this comment

mattip commented Oct 4, 2021

gaborbernat left a comment

Choose a reason for hiding this comment

mattip commented Oct 4, 2021

gaborbernat commented Oct 4, 2021

mattip commented Oct 4, 2021

gaborbernat commented Oct 4, 2021

mattip commented Oct 4, 2021

gaborbernat commented Oct 4, 2021 • edited Loading

mattip commented Oct 4, 2021 • edited by gaborbernat Loading

gaborbernat commented Oct 5, 2021 • edited Loading

mattip commented Oct 5, 2021

mattip commented Oct 5, 2021

gaborbernat commented Oct 5, 2021

mattip commented Oct 5, 2021

mattip commented Oct 5, 2021

gaborbernat commented Oct 5, 2021

mattip commented Oct 6, 2021

mattip commented Oct 6, 2021

gaborbernat commented Oct 6, 2021

mattip commented Oct 12, 2021

mattip commented Oct 12, 2021

mattip commented Oct 12, 2021

mattip commented Oct 15, 2021

mattip commented Oct 18, 2021

rodrigodc commented Oct 18, 2021

mattip commented Oct 4, 2021 •

edited

Loading

mattip commented Oct 4, 2021 •

edited

Loading

gaborbernat commented Oct 4, 2021 •

edited

Loading

mattip commented Oct 4, 2021 •

edited by gaborbernat

Loading

gaborbernat commented Oct 5, 2021 •

edited

Loading