--prefix doesn't honor in-prefix Python

[purajit]

Currently, when --prefix is provided, uv pip install does not respect the in-prefix Python while installing binaries.

In my opinion, in-prefix Python should be respected for two reasons: it feels more per expectations, and
currently, uv requires activating the virtual environment in such cases, which feels antithetical to uv's approach.

I'm down to PR this, but wanted to see what kind of approach would be preferable. Would it be for init to
explicitly set python (unless provided) based on the prefix?

Version: Currently on 0.5.5; I swear that when I was using 0.5.4 it worked in the standard case but not
when VIRTUAL_ENV was set to another path (in which case it still had the same behavior as below); but now
when I try 0.5.4, it errors out for not being able to find a virtual environment.

## default venv
# uv venv
# uv pip install mypy
# head -1 ./.venv/bin/mypy
# ./.venv/bin/mypy -V

## non-default venv without --python
# rm -rf .venv*
# uv venv .venv2
# uv pip install --prefix .venv2 mypy
# head -1 ./.venv2/bin/mypy
# ./.venv2/bin/mypy -V

## non-default venv with --python
# rm -rf .venv*
# uv venv .venv3
# uv pip install --prefix .venv3 --python .venv3 mypy
# ./.venv3/bin/mypy -V

$ which uv
/opt/homebrew/bin/uv

$ uv version
uv 0.5.5 (Homebrew 2024-11-27)

$ uv venv
Using CPython 3.13.0
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

$ uv pip install mypy
Resolved 3 packages in 1ms
Installed 3 packages in 15ms

$ head -1 ./.venv/bin/mypy
#!/Users/purajit.malalur/code/random/uvvenv-prefix-test/.venv/bin/python3

$ which mypy
mypy not found

$ ./.venv/bin/mypy -V
mypy 1.13.0 (compiled: yes)

$ rm -rf .venv*

$ uv venv .venv2
Using CPython 3.13.0
Creating virtual environment at: .venv2
Activate with: source .venv2/bin/activate

$ uv pip install --prefix .venv2 mypy
Using CPython 3.13.0
Resolved 3 packages in 1ms
Installed 3 packages in 17ms

$ head -1 ./.venv2/bin/mypy
#!/Users/purajit.malalur/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/bin/python3.13

$ ./.venv2/bin/mypy -V
Traceback (most recent call last):
  File "/Users/purajit.malalur/code/random/uvvenv-prefix-test/./.venv2/bin/mypy", line 5, in <module>
    from mypy.__main__ import console_entry
ModuleNotFoundError: No module named 'mypy'

$ rm -rf .venv*

$ uv venv .venv3
Using CPython 3.13.0
Creating virtual environment at: .venv3
Activate with: source .venv3/bin/activate

$ uv pip install --prefix .venv3 --python .venv3 mypy
Using CPython 3.13.0 interpreter at: .venv3/bin/python3
Resolved 3 packages in 1ms
Installed 3 packages in 20ms

$ ./.venv3/bin/mypy -V

mypy 1.13.0 (compiled: yes)

Potentially related to 0cd9c54; I can investigate more if this doesn't immediately ring any bells.

[zanieb]

I also suspected #9373. Maybe related to #9371, but I sort of doubt it? Does it reproduce on 0.5.3?

It seems reasonable to find and use the Python in the prefix by default. We should probably ignore it when --system is provided? I'm curious for @charliermarsh's thoughts before someone begins work on it though.

[charliermarsh]

I doubt we ever "discovered" this. I'm unsure on whether we should...

[zanieb]

The idea behind the two linked pull requests is that installing with --prefix is not meant to mutate an existing environment. Perhaps more details about your use-case would be helpful?

[purajit]

Woah you all got to this way faster than I thought! Shouldn't have expected anything less. Love the work
you all do; I've been using uv almost since the beginning and started moving our entire company over to it
starting 0.3.0. It's been so smooth that I get excited every time I find an issue (though nearly always, it's
already been caught and fixed and just updating uv solves it)

more details about your use-case would be helpful

Sure! And I totally understand there's multiple ways to go about this, but basically we have two bioinformatics
tools that need to be installed into their own isolated environment. I considered using uv tool run,
but for various reasons decided not to (changing how the tool is evoked is a stupidly involved change,
and also I currently don't install uv onto any of our production images - we have it only during our build).

The main application is installed into .venv, so I thought about installing the tools into .venv-${tool_name}.
I could absolutely also install it into ${tool_name}/.venv, and then the default uv pip install behavior would
work, but just thought I'd raise this since it still seems like unexpected behavior/inconsistency. I chose
mypy in my example just for demonstration with a popular tool.

Does it reproduce on 0.5.3?

Same error I mentioned about 0.5.4 - I tried a range of versions. It doesn't fail if I first activate
the venv. I'm running the uv binaries straight from release (on Apple Silicon):

DEBUG uv 0.5.3 (56d362208 2024-11-19)
DEBUG Searching for default Python interpreter in virtual environments
DEBUG Found `cpython-3.13.0-macos-aarch64-none` at `/opt/homebrew/bin/python` (search path)
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.13/bin/python3.13`: system interpreter not explicitly requested
DEBUG Found `cpython-3.13.0-macos-aarch64-none` at `/opt/homebrew/bin/python3` (search path)
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.13/bin/python3.13`: system interpreter not explicitly requested
DEBUG Found `cpython-3.11.10-macos-aarch64-none` at `/opt/homebrew/bin/python3.11` (search path)
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.11/bin/python3.11`: system interpreter not explicitly requested
DEBUG Found `cpython-3.12.7-macos-aarch64-none` at `/opt/homebrew/bin/python3.12` (search path)
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.12/bin/python3.12`: system interpreter not explicitly requested
DEBUG Found `cpython-3.13.0-macos-aarch64-none` at `/opt/homebrew/bin/python3.13` (search path)
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.13/bin/python3.13`: system interpreter not explicitly requested
DEBUG Found `cpython-3.9.6-macos-aarch64-none` at `/usr/bin/python3` (search path)
DEBUG Ignoring Python interpreter at `/Library/Developer/CommandLineTools/usr/bin/python3`: system interpreter not explicitly requested
error: No virtual environment found; run `uv venv` to create an environment, or pass `--system` to install into a non-virtual environment

For comparison, this is 5.5's output:

DEBUG uv 0.5.5 (Homebrew 2024-11-27)
DEBUG Searching for default Python interpreter in virtual environments, managed installations, or search path
DEBUG Searching for managed installations at `/Users/purajit.malalur/.local/share/uv/python`
DEBUG Found managed installation `cpython-3.13.0-macos-aarch64-none`
DEBUG Found `cpython-3.13.0-macos-aarch64-none` at `/Users/purajit.malalur/.local/share/uv/python/cpython-3.13.0-macos-aarch64-none/bin/python3.13` (managed installations)
Using CPython 3.13.0
DEBUG Using `--prefix` directory at .venv2
DEBUG Acquired lock for `.venv2`
...

[zanieb]

Thanks for confirming the behavior!

Have you considered uv tool install?

[purajit]

I hadn't! It's a bit different than a standard tool install so it didn't occur to me; we have a lockfile
for these tools since bioinformatics is rather sensitive as an ecosystem (for one of them, we need to
also explicitly install an older setuptools 🙄), so I tested with

UV_TOOL_DIR=<> uv tool install <tool> --with-requirements <lockfile> && \

which worked as expected, and I'll switch over to this.

[zanieb]

Sounds good! With the latest version, you can also provide your lockfile as --constraints.

We're also interested in allowing locks (uv.lock) to be created / used with tool installs.

Could you share any more about the specifics of the tools (I have a bioinformatics background) you're installing and the lockfiles? I'd be curious to hear how it's different than a standard tool install.

[purajit]

Hah - I just got the notification for 0.5.6 and saw the --constraints option :P

I wouldn't say it's too "different", more that I'm so far more used to using uv tool kind of like an
on-the-fly option rather than a way to create a deployable artifact - this is just my perception; I think
once #3560 completes, I'd find it more "native" and obvious for this purpose (specifically, "Tools in a project").

The one slight oddity is Aldy, which requires an installation of setuptools, so the lockfile isn't
just a constraint (I know I can also use uv tool install 'aldy@ git+...' --with setuptools, but I
generally try to avoid having configuration defined as CLI options, and I'd also like to keep it
reproducible with an explicit lockfile, even though in this case it's minor)

[zanieb]

That makes sense. The long-term plan is to allow declarative tool installs, i.e., you'd define them in a TOML file and could install from there. I believe Pixi has a similar feature: https://prefix.dev/blog/pixi_global

[zanieb]

Going to close this as intentional behavior.

[purajit]

@zanieb I didn't fully understand the previous statement, but why is this intentional? Doesn't this mean that installing an executable that imports packages is essentially unsupported with --prefix? I could see potentially worse side-effects too - where the package being imported by an executable within a venv isn't the package installed within that venv.

[zanieb]

Yeah so returning to #9591 (comment) I don't think --prefix is intended to mutate an existing environment and I don't think we should attempt to discover the Python interpreter that exists there. It's sort of a low-level escape hatch and I think it's reasonable to pass --python <path> if you want to opt-in to discovering an existing interpreter.

I closed this because we identified that it wasn't a regression and you had alternative workflows that make more sense. Happy to discuss still though.

Doesn't this mean that installing an executable that imports packages is essentially unsupported with --prefix?

Can you expand on this? I don't quite follow.

[purajit]

Yeah, the alternative works for me and I find it way more idiomatic, so I'm not emotionally/personally
invested in this particular issue anymore, beyond just mentioning my expectations.

Can you expand on this? I don't quite follow.

As in, with the mypy example above, if you're not passing in --python during installation, you can't
actually use the installed mypy.

<took a pause and did some reading; leaving the above so you know what I meant>

Ah, I think I had my interpretation wrong and now the discussions make sense to me - I guess the
intended behavior is to install into a bare directory with the overall directory structure, with
the promises ending there. Reading #3076 and then the PRs above cleared that up for me.
After that, it does become clear that this is absolutely not what I want to use for my case.

This is the help entry for --prefix which confused me; I guess it's referring to just the structure, rather
than the behavior of a virtualenv:

Install packages into lib, bin, and other top-level folders under the specified directory, as if a virtual environment were present at that location

Removing the "virtual environment" wording would be good - or maybe I just read too much into it?

[zanieb]

so I'm not emotionally/personally invested in this particular issue anymore, beyond just mentioning my expectations.

👍

As in, with the mypy example above, if you're not passing in --python during installation, you can't
actually use the installed mypy.

Yeah, I guess I'm not sure if that's within the scope of the intent of --prefix. I've never needed it professionally myself so I can't really speak to that with confidence, I'm just leaning on the context we've established from previous requests.

This is the help entry for --prefix which confused me; ...

I think that's there to explain how the layout differs from --target

❯ uv pip install --target example anyio
Using Python 3.12.7 environment at bar
Resolved 3 packages in 115ms
Installed 3 packages in 5ms
 + anyio==4.6.2.post1
 + idna==3.10
 + sniffio==1.3.1

❯ tree -L1 example
example
├── anyio
├── anyio-4.6.2.post1.dist-info
├── idna
├── idna-3.10.dist-info
├── sniffio
└── sniffio-1.3.1.dist-info

7 directories, 0 files

We certainly could adjust the wording though.

Did

In general, prefer the use of --python to install into an alternate environment, as scripts and other artifacts installed via --prefix will reference the installing interpreter, rather than any interpreter added to the --prefix directory, rendering them non-portable.

not give you pause? :) Perhaps we should expand that warning?

[purajit]

not give you pause? :) Perhaps we should expand that warning?

I'm gonna be completely honest - when I checked the reference I didn't notice the second paragraph. Completely
by bad. That's a crystal-clear entry that explains exactly what I needed. I'm so sorry.

[zanieb]

No problem! Thanks for taking the time to explain everything.