Creating venvs via standard library's venv.create(..., with_pip=True) fails for binaries linked against libpython dynamically

[smheidrich]

Minimal example:

import subprocess
import venv
from tempfile import mkdtemp
from traceback import print_exc

tmp_dir_path = mkdtemp(prefix="venv-create-bug-example.")
print(f"creating venv in: {tmp_dir_path}")
try:
  venv.create(tmp_dir_path, with_pip=True)
except subprocess.CalledProcessError as e:
  print_exc()
  print(f"subprocess stdout:\n{e.stdout}")
  print(f"subprocess stderr:\n{e.stderr}")
  exit(1)

Running this with a python-build-standalone binary that has a shared library dependency on libpython*.so, e.g. cpython-3.12.7+20241016-x86_64-unknown-linux-gnu-install_only will make this fail with (extra newlines inserted for readability):

creating venv in: /tmp/venv-create-bug-example.pj058eb3
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command
'['/tmp/venv-create-bug-example.pj058eb3/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']'
returned non-zero exit status 127.
subprocess stdout:
b'/tmp/venv-create-bug-example.pj058eb3/bin/python:
error while loading shared libraries:
/tmp/venv-create-bug-example.pj058eb3/bin/../lib/libpython3.12.so.1.0:
cannot open shared object file: No such file or directory\n'
...

Obviously, this is because the python binary links against libpython*.so dynamically with a relative path, and there no longer is a libpython*.so at this path when the Python executable (or a symlink to it) finds itself in the venv.

See also the corresponding issue in uv:

• libpython missing astral-sh/uv#6812

[willnode]

Maybe helps

$ cd ~/.pyenv/versions/3.11.10
$ ld bin/python
ld: warning: $ORIGIN/../lib/libpython3.11.so.1.0, needed by bin/python, not found (try using -rpath or -rpath-link)
ld: warning: cannot find entry symbol _start; not setting start address
ld: bin/python: undefined reference to `Py_BytesMain'
$ ln -s lib ../lib
$ export LD_LIBRARY_PATH=~/.pyenv/versions/3.11.10:$LD_LIBRARY_PAT
$ ld bin/python
ld: warning: cannot find entry symbol _start; not setting start address

Adding LD_LIBRARY_PATH works so...

// your test file
$ nano x.py
$ python x.py 
creating venv in: /tmp/venv-create-bug-example.fg3ebyf5
$