Python venv "python3.XX" links to base Python, not the Python binary wrapper

[mcdonc]

This began life as #901, but it deserves its own bug.

python-rewrite branch.

When languages.python.venv.enable = true;, the resulting venv has a Python that points at the base Python instead of the binary wrapper created within python.nix, even though the executable which is run to create the venv is indeed the wrapper. This is a problem, because it defeats the purpose of languages.python.manylinux.enable and languages.python.libraries.

I have not been able to convince venv itself to link to the wrapper. I attempted to just change the bin/python3.11 symlink in the venv so it points at the wrapper, but then when I try to use pip:

❯ devenv shell
• Building shell ...
✔ Building shell in 2.5s.
• Entering shell
Python interpreter changed, rebuilding Python venv...
/nix/store/ricl9i2v2707nmms9hf7xzfqsyf0xmw2-python3-3.11.6-env/bin/python3.11 -m venv --upgrade-deps /home/chrism/projects/devenv/tmp/.devenv/state/venv
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp]$ cd .devenv/state/venv/bin/
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp/.devenv/state/venv/bin]$ ln -sf /nix/store/ricl9i2v2707nmms9hf7xzfqsyf0xmw2-python3-3.11.6-env/bin/python3.11 python3.11
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp/.devenv/state/venv/bin]$ cat pip
#!/home/chrism/projects/devenv/tmp/.devenv/state/venv/bin/python3.11
# -*- coding: utf-8 -*-
import re
import sys
from pip._internal.cli.main import main
if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp/.devenv/state/venv/bin]$ pip install numpy
Traceback (most recent call last):
  File "/home/chrism/projects/devenv/tmp/.devenv/state/venv/bin/pip", line 5, in <module>
    from pip._internal.cli.main import main
ModuleNotFoundError: No module named 'pip'

If I force pip to be installed when the wrapper is created, it produces a pip that tries to install to the Nix store somehow.

chrism@optinix:~/projects/devenv/tmp]$ cd .devenv/state/venv/bin/
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp/.devenv/state/venv/bin]$ ln -sf /nix/store/7cig2xr46p06cqr0manygbnnygq86n1c-python3-3.11.6-env/bin/python3.11 python3.11
(devenv) (venv) 
[chrism@optinix:~/projects/devenv/tmp/.devenv/state/venv/bin]$ pip install numpy
Collecting numpy
  Obtaining dependency information for numpy from https://files.pythonhosted.org/packages/5a/62/007b63f916aca1d27f5fede933fda3315d931ff9b2c28b9c2cf388cd8edb/numpy-1.26.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata
  Using cached numpy-1.26.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Using cached numpy-1.26.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.3 MB)
Installing collected packages: numpy
ERROR: Could not install packages due to an OSError: [Errno 30] Read-only file system: '/nix/store/7cig2xr46p06cqr0manygbnnygq86n1c-python3-3.11.6-env/lib/python3.11/site-packages/numpy-1.26.3.dist-info'

Some other helpful info may be in:

https://discourse.nixos.org/t/how-to-build-python-virtualenv-with-packages-provided-by-python3-withpackages/24766

[mcdonc]

Of course this in devenv.nix does get the job done (but would inevitably screw with other programs):

  env.LD_LIBRARY_PATH = with pkgs; lib.makeLibraryPath [
    stdenv.cc.cc.lib
    pythonManylinuxPackages.manylinux2014Package
    zlib
  ];

[thenonameguy]

Hi @mcdonc , can you try pointing your nixpkgs input in devenv.yaml to this branch?
NixOS/nixpkgs#275701

It should in theory fix the problem.

[mcdonc]

Hi @mcdonc , can you try pointing your nixpkgs input in devenv.yaml to this branch? NixOS/nixpkgs#275701

It should in theory fix the problem.

It does! Nice patch, I was tearing my hair out :)

[domenkozar]

We now use https://github.com/cachix/devenv-nixpkgs with the patch, so this is fixed