Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Toolchain::find_or_fetch and use in uv venv --preview #4138

Merged
merged 1 commit into from
Jun 10, 2024

Conversation

zanieb
Copy link
Member

@zanieb zanieb commented Jun 7, 2024

Extends #4121
Part of #2607

Adds support for managed toolchain fetching to uv venv, e.g.

❯ cargo run -q -- venv --python 3.9.18 --preview -v
DEBUG Searching for Python 3.9.18 in search path or managed toolchains
DEBUG Searching for managed toolchains at `/Users/zb/Library/Application Support/uv/toolchains`
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Found CPython 3.9.6 at `/usr/bin/python3` (search path)
DEBUG Found CPython 3.12.3 at `/opt/homebrew/bin/python3` (search path)
DEBUG Requested Python not found, checking for available download...
DEBUG Using registry request timeout of 30s
INFO Fetching requested toolchain...
DEBUG Downloading https://github.com/indygreg/python-build-standalone/releases/download/20240224/cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst to temporary location /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp
DEBUG Extracting cpython-3.9.18%2B20240224-aarch64-apple-darwin-pgo%2Blto-full.tar.zst
DEBUG Moving /Users/zb/Library/Application Support/uv/toolchains/.tmpgohKwp/python to /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none
Using Python 3.9.18 interpreter at: /Users/zb/Library/Application Support/uv/toolchains/cpython-3.9.18-macos-aarch64-none/install/bin/python3
Creating virtualenv at: .venv
INFO Removing existing directory
Activate with: source .venv/bin/activate

The preview flag is required. The fetch is performed if we can't find an interpreter that satisfies the request. Once fetched, the toolchain will be available for later invocations that include the --preview flag. There will be follow-ups to improve toolchain management in general, there is still outstanding work from the initial implementation.

@zanieb zanieb added the preview Experimental behavior label Jun 7, 2024
Comment on lines -94 to -126
// Order matters here, as we overwrite previous links
info!("Installing to `{}`...", toolchain_dir.user_display());

// On Windows, linking the executable generally results in broken installations
// and each toolchain path will need to be added to the PATH separately in the
// desired order
#[cfg(unix)]
{
let mut links: HashMap<PathBuf, PathBuf> = HashMap::new();
for (version, path) in results {
// TODO(zanieb): This path should be a part of the download metadata
let executable = path.join("install").join("bin").join("python3");
for target in [
toolchain_dir.join(format!("python{}", version.python_full_version())),
toolchain_dir.join(format!("python{}.{}", version.major(), version.minor())),
toolchain_dir.join(format!("python{}", version.major())),
toolchain_dir.join("python"),
] {
// Attempt to remove it, we'll fail on link if we couldn't remove it for some reason
// but if it's missing we don't want to error
let _ = fs::remove_file(&target);
symlink(&executable, &target).await?;
links.insert(target, executable.clone());
}
}
for (target, executable) in links.iter().sorted() {
info!(
"Linked `{}` to `{}`",
target.user_display(),
executable.user_display()
);
}
};
Copy link
Member Author

@zanieb zanieb Jun 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This whole thing is no longer necessary, we've been discovering managed toolchains via their directories instead of executables for a while now

@zanieb zanieb marked this pull request as ready for review June 7, 2024 18:15
zanieb added a commit that referenced this pull request Jun 7, 2024
In preparation for managed toolchains #2607, just renames the crate to
something broader.

See #4121 and #4138 to see the final
intent.
zanieb added a commit that referenced this pull request Jun 7, 2024
Extends #4120 
Part of #2607 

There should be no behavior changes here. Restructures the discovery API
to be focused on a toolchain first perspective in preparation for
exposing a `find_or_fetch` method for toolchains in
#4138.
@zanieb zanieb force-pushed the zb/toolchain-iiii branch from 350ebab to 948cd64 Compare June 7, 2024 19:20
Base automatically changed from zb/toolchain-iiii to main June 7, 2024 19:29
@zanieb zanieb force-pushed the zb/toolchain-v branch 2 times, most recently from 62c4cad to b476a30 Compare June 7, 2024 23:32
@zanieb zanieb requested review from charliermarsh and konstin June 7, 2024 23:32
Copy link
Member

@charliermarsh charliermarsh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will review, but while it's on my mind: should there be a way to force a project to use managed toolchains? Like, fetch and install even if you have some non-managed thing on your PATH?

crates/uv-toolchain/src/downloads.rs Show resolved Hide resolved
crates/uv-toolchain/src/downloads.rs Outdated Show resolved Hide resolved
crates/uv-toolchain/src/downloads.rs Outdated Show resolved Hide resolved
@zanieb
Copy link
Member Author

zanieb commented Jun 9, 2024

Will review, but while it's on my mind: should there be a way to force a project to use managed toolchains? Like, fetch and install even if you have some non-managed thing on your PATH?

There's a hidden environment variable right now (UV_FORCE_MANAGED_PYTHON) but yeah I think more holistically we're going to have to expose some enum option that more clearly describes the allowed toolchain. Managed toolchains are grouped in with "system" Python installations right now which isn't very clear.

crates/uv-toolchain/src/downloads.rs Outdated Show resolved Hide resolved
crates/uv-toolchain/src/downloads.rs Outdated Show resolved Hide resolved
crates/uv-toolchain/src/managed.rs Outdated Show resolved Hide resolved
Ok(Self { root: root.into() })
}

/// Prefer, in order:
/// 1. The specific toolchain directory specified by the user, i.e., `UV_TOOLCHAIN_DIR`
/// 2. A directory in the system-appropriate user-level data directory, e.g., `~/.local/uv/toolchains`
/// 3. A directory in the local data directory, e.g., `./.uv/toolchains`
pub fn from_settings() -> Result<Self, io::Error> {
pub fn from_settings() -> Result<Self, Error> {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the #[error(transparent)] on Error, i think the callsites of from_settings should have context that the error comes from toolchain discovery.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what the action item is here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the suggestion is that if you're using transparent IO errors, there will be no indication of why the IO error occurred. Like, if we fail to read or write a file, the entire error trace would just be "Failed to write the file". Using context, you can say that we failed during toolchain discovery.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah interesting, thanks. I'll look into that.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's two kinds of adding context to errors: One would be replacing something like

#[error(transparent)]
ManagedToolchain(#[from] crate::managed::Error),

with

#[error("Something about what this error is, maybe a {python version}")]
ManagedToolchain(#[from] crate::managed::Error),

The other is replacing InstalledToolchains::from_settings()?.init()? with

let toolchains = InstalledToolchains::from_settings()?.init()?;
let toolchains = InstalledToolchains::from_settings()
    .context("Failed to load installed toolchains")?
    .init()
    .context("Failed to initialize toolchain discovery")?;

This requires some manual auditing of who adds the context, e.g. if you're asking for a specific python version, either the caller or the callee should be responsible for reporting the python version.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks again!

# Conflicts:
#	crates/uv-toolchain/src/toolchain.rs
@zanieb zanieb enabled auto-merge (squash) June 10, 2024 14:07
@zanieb zanieb merged commit 45df889 into main Jun 10, 2024
47 checks passed
@zanieb zanieb deleted the zb/toolchain-v branch June 10, 2024 14:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
preview Experimental behavior
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants