ModuleNotFoundError in pypy3.8 created with symlinked binary

[starhel]

Issue
I'm keeping all my python versions in $HOME/bin directory as symlinks. From 20.16.7 pypy3 virtualenvs created this way are not working properly:

> python
ModuleNotFoundError: No module named 'encodings'
debug: OperationError:
debug:  operror-type: ModuleNotFoundError
debug:  operror-value: No module named 'encodings'

I've noticed that the home key in pyvenv.cfg changed from a directory where symlinked python exists to a directory with a symlink. Changing this value to correct one fixed the issue with pypy.

Environment

• OS: Ubuntu 20
• pip list of the venv with virtualenv installed:

Package      Version
------------ -------
distlib      0.3.6
filelock     3.8.0
pip          20.2.3
platformdirs 2.5.4
setuptools   49.2.1
virtualenv   20.16.7

Reproduction

wget https://downloads.python.org/pypy/pypy3.8-v7.3.9-linux64.tar.bz2
tar -xf pypy3.8-v7.3.9-linux64.tar.bz2 
mkdir bin
ln -s ../pypy3.8-v7.3.9-linux64/bin/pypy3.8 bin/pypy3.8

python3 -m venv venv
source venv/bin/activate
pip install --upgrade virtualenv

virtualenv --always-copy -p bin/pypy3.8 venv_pypy
source venv_pypy/bin/activate
python --version  # error
# change home in pyvenv.cfg from bin to pypy3.8-v7.3.9-linux64
python --version  # ok

Output of the virtual environment creation

51 setup logging to NOTSET [DEBUG report:35]
55 find interpreter for spec PythonSpec(path=bin/pypy3.8) [INFO builtin:56]
56 Attempting to acquire lock 140514597215584 on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:172]
56 Lock 140514597215584 acquired on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:176]
56 got python info of /home/astachlewski/tmp/bin/pypy3.8 from /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.json [DEBUG via_disk_folder:129]
57 Attempting to release lock 140514597215584 on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:209]
57 Lock 140514597215584 released on /home/astachlewski/.local/share/virtualenv/py_info/1/42185fae4a9759b8ee9b49fdff8b66ca18709e3cfc13aa4daa779c8cf7116d3e.lock [DEBUG _api:212]
57 proposed PythonInfo(spec=PyPy3.8.13.final.0-64, exe=/home/astachlewski/tmp/bin/pypy3.8, platform=linux, version='3.8.13 (4b1398fe9d76ad762155d03684c2a153d230b2ef, Mar 29 2022, 07:08:24)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [INFO builtin:63]
57 accepted PythonInfo(spec=PyPy3.8.13.final.0-64, exe=/home/astachlewski/tmp/bin/pypy3.8, platform=linux, version='3.8.13 (4b1398fe9d76ad762155d03684c2a153d230b2ef, Mar 29 2022, 07:08:24)\n[PyPy 7.3.9 with GCC 10.2.1 20210130 (Red Hat 10.2.1-11)]', encoding_fs_io=utf-8-UTF-8) [DEBUG builtin:65]
58 filesystem is case-sensitive [DEBUG info:24]
79 create virtual environment via PyPy3Posix(dest=/home/astachlewski/tmp/venv_pypy, clear=False, no_vcs_ignore=False, global=False) [INFO session:48]
80 create folder /home/astachlewski/tmp/venv_pypy/bin [DEBUG _sync:9]
80 create folder /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages [DEBUG _sync:9]
80 write /home/astachlewski/tmp/venv_pypy/pyvenv.cfg [DEBUG pyenv_cfg:30]
80 	home = /home/astachlewski/tmp/bin [DEBUG pyenv_cfg:34]
80 	implementation = PyPy [DEBUG pyenv_cfg:34]
80 	version_info = 3.8.13.final.0 [DEBUG pyenv_cfg:34]
80 	virtualenv = 20.16.7 [DEBUG pyenv_cfg:34]
80 	include-system-site-packages = false [DEBUG pyenv_cfg:34]
80 	base-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
80 	base-exec-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
80 	base-executable = /home/astachlewski/tmp/bin/pypy3.8 [DEBUG pyenv_cfg:34]
80 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/pypy [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/pypy3 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python3 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/bin/pypy3.8 to /home/astachlewski/tmp/venv_pypy/bin/python3.8 [DEBUG _sync:36]
81 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/bin/libpypy3-c.so.debug to /home/astachlewski/tmp/venv_pypy/bin/libpypy3-c.so.debug [DEBUG _sync:36]
94 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/bin/libpypy3-c.so to /home/astachlewski/tmp/venv_pypy/bin/libpypy3-c.so [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libncursesw.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libncursesw.so.6 [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtinfow.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libtinfow.so.6 [DEBUG _sync:36]
128 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtk8.5.so to /home/astachlewski/tmp/venv_pypy/lib/libtk8.5.so [DEBUG _sync:36]
129 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libpanelw.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libpanelw.so.6 [DEBUG _sync:36]
129 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libexpat.so.1 to /home/astachlewski/tmp/venv_pypy/lib/libexpat.so.1 [DEBUG _sync:36]
130 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/liblzma.so.5 to /home/astachlewski/tmp/venv_pypy/lib/liblzma.so.5 [DEBUG _sync:36]
130 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libgdbm.so.4 to /home/astachlewski/tmp/venv_pypy/lib/libgdbm.so.4 [DEBUG _sync:36]
130 copy directory /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/tcl8.5 to /home/astachlewski/tmp/venv_pypy/lib/tcl8.5 [DEBUG _sync:36]
143 copy directory /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/tk8.5 to /home/astachlewski/tmp/venv_pypy/lib/tk8.5 [DEBUG _sync:36]
152 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libsqlite3.so.0 to /home/astachlewski/tmp/venv_pypy/lib/libsqlite3.so.0 [DEBUG _sync:36]
153 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libffi.so.6 to /home/astachlewski/tmp/venv_pypy/lib/libffi.so.6 [DEBUG _sync:36]
153 copy /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64/lib/libtcl8.5.so to /home/astachlewski/tmp/venv_pypy/lib/libtcl8.5.so [DEBUG _sync:36]
154 create virtualenv import hook file /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_virtualenv.pth [DEBUG api:89]
154 create /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_virtualenv.py [DEBUG api:92]
154 ============================== target debug ============================== [DEBUG session:50]
154 debug via /home/astachlewski/tmp/venv_pypy/bin/pypy3 /home/astachlewski/tmp/venv/lib/python3.8/site-packages/virtualenv/create/debug.py [DEBUG creator:197]
154 {
  "out": "",
  "err": "ModuleNotFoundError: No module named 'encodings'\ndebug: OperationError:\ndebug:  operror-type: ModuleNotFoundError\ndebug:  operror-value: No module named 'encodings'\n",
  "returncode": 1,
  "exception": "Exception(\"ModuleNotFoundError: No module named 'encodings'\\ndebug: OperationError:\\ndebug:  operror-type: ModuleNotFoundError\\ndebug:  operror-value: No module named 'encodings'\\n\")"
} [DEBUG session:51]
194 add seed packages via FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/astachlewski/.local/share/virtualenv) [INFO session:55]
198 got embed update of distribution setuptools from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/setuptools.json [DEBUG via_disk_folder:129]
199 got embed update of distribution wheel from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/wheel.json [DEBUG via_disk_folder:129]
199 got embed update of distribution pip from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/pip.json [DEBUG via_disk_folder:129]
206 got embed update of distribution wheel from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/wheel.json [DEBUG via_disk_folder:129]
206 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/wheel-0.37.1-py2.py3-none-any.whl [DEBUG periodic_update:41]
206 got embed update of distribution setuptools from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/setuptools.json [DEBUG via_disk_folder:129]
206 got embed update of distribution pip from /home/astachlewski/.local/share/virtualenv/wheel/3.8/embed/3/pip.json [DEBUG via_disk_folder:129]
207 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/setuptools-65.5.0-py3-none-any.whl [DEBUG periodic_update:41]
207 using periodically updated wheel /home/astachlewski/.local/share/virtualenv/wheel/house/pip-22.3-py3-none-any.whl [DEBUG periodic_update:41]
208 install wheel from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/wheel-0.37.1-py2.py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
208 install setuptools from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/setuptools-65.5.0-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
209 Attempting to acquire lock 140514568567536 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:172]
209 Attempting to acquire lock 140514568567296 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:172]
209 install pip from wheel /home/astachlewski/.local/share/virtualenv/wheel/house/pip-22.3-py3-none-any.whl via CopyPipInstall [DEBUG via_app_data:47]
209 Lock 140514568567536 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:176]
210 Attempting to acquire lock 140514568567632 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:172]
210 Lock 140514568567296 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:176]
210 Lock 140514568567632 acquired on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:176]
210 Attempting to release lock 140514568567536 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:209]
210 Lock 140514568567536 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any.lock [DEBUG _api:212]
210 Attempting to release lock 140514568567632 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:209]
210 Lock 140514568567632 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any.lock [DEBUG _api:212]
210 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip-22.3.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip-22.3.dist-info [DEBUG _sync:36]
210 Attempting to release lock 140514568567296 on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:209]
211 Lock 140514568567296 released on /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any.lock [DEBUG _api:212]
211 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel-0.37.1.virtualenv [DEBUG _sync:36]
212 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools-65.5.0.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools-65.5.0.dist-info [DEBUG _sync:36]
212 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel [DEBUG _sync:36]
214 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip [DEBUG _sync:36]
217 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/pkg_resources to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pkg_resources [DEBUG _sync:36]
225 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/wheel-0.37.1-py2.py3-none-any/wheel-0.37.1.dist-info to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/wheel-0.37.1.dist-info [DEBUG _sync:36]
231 generated console scripts wheel-3.8 wheel3 wheel3.8 wheel [DEBUG base:41]
241 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools [DEBUG _sync:36]
271 copy directory /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/_distutils_hack to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/_distutils_hack [DEBUG _sync:36]
271 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/distutils-precedence.pth to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/distutils-precedence.pth [DEBUG _sync:36]
271 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/setuptools-65.5.0-py3-none-any/setuptools-65.5.0.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/setuptools-65.5.0.virtualenv [DEBUG _sync:36]
272 generated console scripts  [DEBUG base:41]
288 copy /home/astachlewski/.local/share/virtualenv/wheel/3.8/image/1/CopyPipInstall/pip-22.3-py3-none-any/pip-22.3.virtualenv to /home/astachlewski/tmp/venv_pypy/lib/pypy3.8/site-packages/pip-22.3.virtualenv [DEBUG _sync:36]
289 generated console scripts pip3 pip pip3.8 pip-3.8 [DEBUG base:41]
289 add activators for Bash, CShell, Fish, Nushell, PowerShell, Python [INFO session:61]
292 write /home/astachlewski/tmp/venv_pypy/pyvenv.cfg [DEBUG pyenv_cfg:30]
292 	home = /home/astachlewski/tmp/bin [DEBUG pyenv_cfg:34]
292 	implementation = PyPy [DEBUG pyenv_cfg:34]
292 	version_info = 3.8.13.final.0 [DEBUG pyenv_cfg:34]
292 	virtualenv = 20.16.7 [DEBUG pyenv_cfg:34]
292 	include-system-site-packages = false [DEBUG pyenv_cfg:34]
292 	base-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
293 	base-exec-prefix = /home/astachlewski/tmp/pypy3.8-v7.3.9-linux64 [DEBUG pyenv_cfg:34]
293 	base-executable = /home/astachlewski/tmp/bin/pypy3.8 [DEBUG pyenv_cfg:34]
293 created virtual environment PyPy3.8.13.final.0-64 in 243ms
  creator PyPy3Posix(dest=/home/astachlewski/tmp/venv_pypy, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/astachlewski/.local/share/virtualenv)
    added seed packages: pip==22.3, setuptools==65.5.0, wheel==0.37.1
  activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator [WARNING __main__:17]

[gaborbernat]

This is probably related to the changes introduced by @vfazio?

[vfazio]

oof. i'm not familiar with pypy at all but the driving force of #2441 was due to PEP405 which explicitly states that the "home" key should be the parent of the python binary: https://peps.python.org/pep-0405/#specification

CPython handles this by recursing up the directories for landmark directories/files: https://github.com/python/cpython/blob/main/Modules/getpath.py#L208

Sorry that my PR broke functionality. I'm open to ideas here.

[vfazio]

Note that the "correct" value for home is important for 3.11+ since the home key now drives path resolution on posix for the system executable (where prior to 3.11 it did not). See the following:

python-poetry/poetry#6940
python/cpython#99204

https://bugs.python.org/issue46028
python/cpython#30144

[starhel]

I've checked also venv module in pypy3.8 and it's also not working with symlinked binary and --copies file.

> ./bin/pypy3.8 -m venv venv_pypy --copies
Error: Command '['/home/astachlewski/tmp/venv_pypy/bin/pypy3.8', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 127.

Both virtualenv and venv work if binaries are symlinked instead of copied.

As far as I understand all these linked discussions and PEP405, I think that virtualenv is working correctly. My workaround is probably a not supported hack (I've added this bin directory to PATH to provide all pypy and cpython versions without colissions -> adding original bin paths leads to cpython be shadowed by pypy).

[gaborbernat]

Closing per above, if upstream pypy changes behaviour we can revisit.