Discover and prefer the parent interpreter when invoked with `python -m uv` #3736

zanieb · 2024-05-22T14:12:33Z

Closes #2222
Closes #2058
Replaces #2338
See also #2649

We use an environment variable (UV_INTERNAL__PARENT_INTERPRETER) to track the invoking interpreter when python -m uv is used. The parent interpreter is preferred over all other sources (though it will be skipped if it does not meet a --python request or if --system is used and it belongs to a virtual environment). We warn if --system is not provided and this interpreter would mutate system packages, but allow it.

crates/uv-interpreter/src/environment.rs

zanieb · 2024-05-22T14:20:24Z

crates/uv-interpreter/src/discovery.rs

@@ -1053,22 +1063,16 @@ impl SourceSelector {
                        InterpreterSource::SearchPath,
                        InterpreterSource::PyLauncher,
                        InterpreterSource::ManagedToolchain,
+                        InterpreterSource::ParentInterpreter,


This is in the SystemPython::Required clause. Maybe I should move it out? This may be a bit weird if you do ./.venv/bin/python -m uv pip install --system as we'll still find this Python... but also a bit weird if you do /sys/python -m uv --system and we find a different Python?

I think I can fix this with some changes to discovery

Is the ideal behavior that:

./.venv/bin/python -m uv pip install --system does not find .venv/bin/python

/sys/python -m uv --system does find /sys/python

?

I think so yea. With #3739 this will be possible

(rebasing this onto that now)

The behavior should be as described now, I'll do some QA.

konstin · 2024-05-22T15:07:25Z

python/uv/__main__.py

@@ -31,6 +31,9 @@ def _run() -> None:
    if venv:
        env.setdefault("VIRTUAL_ENV", venv)

+    # Let `uv` know that it was spawned by this Python interpreter
+    env["UV__PARENT_INTERPRETER"] = sys.executable


Can we use something other than the double underscore, like a name that is so long that no one likes to use it outside that script?

I feel like the double underscore is sufficient for saying it's not user facing and we're not going to document or support it elsewhere. Idk about making the name arbitrarily longer.... I guess I could do UV_PRIVATE__...?

Added more suffix

Previously, we enforced `SystemPython` outside of the interpreter discovery exclusively with source selection. Now, we perform additional filtering of interpreters depending on if they are a virtual environment. This should not change any existing behavior, but will make it much easier to have consistent behavior in ambiguous cases like #3736 (comment) where a source could provide either a system interpreter or virtual environment interpreter.

…-m uv ...`

zanieb · 2024-05-22T16:23:44Z

crates/uv-interpreter/src/discovery.rs

+    /// Only allow a system Python if passed directly i.e. via [`InterpreterSource::ProvidedPath`] or [`InterpreterSource::ParentInterpreter`]
    #[default]
+    Explicit,
+    /// Do not allow a system Python
    Disallowed,


I decided to add an Explicit variant to avoid confusing internal behavior where SystemPython::Disallowed could still get a system Python from this environment variable.

# Conflicts: # crates/uv-interpreter/src/discovery.rs

zanieb · 2024-05-22T16:28:40Z

e.g.

❯ RUST_LOG=uv=trace ./uv/.venv/bin/python -m uv -v pip install anyio
TRACE Cached interpreter info for Python 3.12.3, skipping probing: uv/.venv/bin/python
TRACE Found Python interpreter cpython 3.12.3 at /Users/zb/workspace/uv/.venv/bin/python from parent interpreter
DEBUG Using Python 3.12.3 environment at uv/.venv/bin/python
DEBUG Trying to lock if free: uv/.venv/.lock
TRACE Comparing installed with source: Registry(InstalledRegistryDist { name: PackageName("anyio"), version: "4.3.0", path: "/Users/zb/workspace/uv/.venv/lib/python3.12/site-packages/anyio-4.3.0.dist-info" }) Registry { specifier: VersionSpecifiers([]), index: None }
TRACE Comparing installed with source: Registry(InstalledRegistryDist { name: PackageName("sniffio"), version: "1.3.1", path: "/Users/zb/workspace/uv/.venv/lib/python3.12/site-packages/sniffio-1.3.1.dist-info" }) Registry { specifier: VersionSpecifiers([VersionSpecifier { operator: GreaterThanEqual, version: "1.1" }]), index: None }
TRACE Comparing installed with source: Registry(InstalledRegistryDist { name: PackageName("idna"), version: "3.7", path: "/Users/zb/workspace/uv/.venv/lib/python3.12/site-packages/idna-3.7.dist-info" }) Registry { specifier: VersionSpecifiers([VersionSpecifier { operator: GreaterThanEqual, version: "2.8" }]), index: None }
DEBUG Requirement satisfied: anyio
DEBUG Requirement satisfied: idna>=2.8
DEBUG Requirement satisfied: sniffio>=1.1
DEBUG All editables satisfied: 
Audited 1 package in 1ms

❯ RUST_LOG=uv=trace ./uv/.venv/bin/python -m uv -v pip install anyio --system
TRACE Searching PATH for executables: python3, python
TRACE Checking `PATH` directory for interpreters: /Users/zb/bin
TRACE Checking `PATH` directory for interpreters: /Users/zb/.local/bin
TRACE Checking `PATH` directory for interpreters: /Users/zb/.cargo/bin
TRACE Checking `PATH` directory for interpreters: /opt/homebrew/bin
TRACE Found possible Python executable: /opt/homebrew/bin/python3
TRACE Cached interpreter info for Python 3.12.3, skipping probing: /opt/homebrew/bin/python3
TRACE Found Python interpreter cpython 3.12.3 at /opt/homebrew/bin/python3 from search path
DEBUG Using Python 3.12.3 environment at /opt/homebrew/opt/python@3.12/bin/python3.12
error: The interpreter at /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12 is externally managed, and indicates the following:

❯ ./uv/.venv/bin/python -m uv -v pip install anyio --python 3.10
INFO Found active virtual environment (via VIRTUAL_ENV) at: /Users/zb/workspace/uv/.venv
DEBUG Ignoring Python interpreter at `/Users/zb/workspace/uv/bin/cpython-3.10.13-macos-aarch64-none/install/bin/python3`: system intepreter not explicit
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.12/bin/python3.12`: system intepreter not explicit
DEBUG Ignoring Python interpreter at `/Library/Developer/CommandLineTools/usr/bin/python3`: system intepreter not explicit
DEBUG Ignoring Python interpreter at `/opt/homebrew/opt/python@3.12/bin/python3.12`: system intepreter not explicit
error: No interpreter found for Python 3.10 in all sources

❯ /opt/homebrew/bin/python3 -m uv -v pip install anyio
DEBUG Allowing system Python interpreter at `/opt/homebrew/opt/python@3.12/bin/python3.12`
DEBUG Using Python 3.12.3 environment at /opt/homebrew/opt/python@3.12/bin/python3.12

zanieb added the enhancement New feature or improvement to existing functionality label May 22, 2024

zanieb changed the title ~~Discover and prefer the parent interpreter when invoked with python -m uv ...~~ Discover and prefer the parent interpreter when invoked with python -m uv May 22, 2024

zanieb commented May 22, 2024

View reviewed changes

crates/uv-interpreter/src/environment.rs Outdated Show resolved Hide resolved

zanieb marked this pull request as ready for review May 22, 2024 14:18

zanieb commented May 22, 2024

View reviewed changes

konstin approved these changes May 22, 2024

View reviewed changes

zanieb mentioned this pull request May 22, 2024

Filter discovered Python interpreters by system preference #3739

Merged

zanieb force-pushed the zb/parent-interp branch from b8fcf7c to a482251 Compare May 22, 2024 16:05

zanieb changed the base branch from main to zb/filter-system-preferences May 22, 2024 16:05

zanieb force-pushed the zb/parent-interp branch from a482251 to d6bc390 Compare May 22, 2024 16:18

Base automatically changed from zb/filter-system-preferences to main May 22, 2024 16:22

Discover and prefer the parent interpreter when invoked with `python …

dfeccb6

…-m uv ...`

zanieb force-pushed the zb/parent-interp branch from d6bc390 to 509156b Compare May 22, 2024 16:22

zanieb commented May 22, 2024

View reviewed changes

Add SystemPython::Explicit

34a7fa5

# Conflicts: # crates/uv-interpreter/src/discovery.rs

zanieb force-pushed the zb/parent-interp branch from 509156b to 34a7fa5 Compare May 22, 2024 16:24

charliermarsh approved these changes May 22, 2024

View reviewed changes

zanieb merged commit 5fe8910 into main May 22, 2024
44 checks passed

zanieb deleted the zb/parent-interp branch May 22, 2024 16:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Discover and prefer the parent interpreter when invoked with `python -m uv` #3736

Discover and prefer the parent interpreter when invoked with `python -m uv` #3736

zanieb commented May 22, 2024 •

edited

Loading

zanieb May 22, 2024

zanieb May 22, 2024

charliermarsh May 22, 2024

zanieb May 22, 2024

zanieb May 22, 2024

zanieb May 22, 2024

konstin May 22, 2024

zanieb May 22, 2024

zanieb May 22, 2024

zanieb May 22, 2024

zanieb commented May 22, 2024 •

edited

Loading

Discover and prefer the parent interpreter when invoked with python -m uv #3736

Discover and prefer the parent interpreter when invoked with python -m uv #3736

Conversation

zanieb commented May 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb commented May 22, 2024 • edited Loading

Discover and prefer the parent interpreter when invoked with `python -m uv` #3736

Discover and prefer the parent interpreter when invoked with `python -m uv` #3736

zanieb commented May 22, 2024 •

edited

Loading

zanieb commented May 22, 2024 •

edited

Loading