Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add UV_CONCURRENT_INSTALLS variable in favor of RAYON_NUM_THREADS #3646

Merged
merged 2 commits into from
May 18, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -560,6 +560,8 @@ uv accepts the following command-line arguments as environment variables:
will perform at any given time.
- `UV_CONCURRENT_BUILDS`: Sets the maximum number of source distributions that `uv` will build
concurrently at any given time.
- `UV_CONCURRENT_INSTALLS`: Used to control the number of threads used when installing and unzipping
packages.

In each case, the corresponding command-line argument takes precedence over an environment variable.

Expand All @@ -583,8 +585,6 @@ In addition, uv respects the following environment variables:
- `FISH_VERSION`: Used to detect the use of the Fish shell.
- `BASH_VERSION`: Used to detect the use of the Bash shell.
- `ZSH_VERSION`: Used to detect the use of the Zsh shell.
- `RAYON_NUM_THREADS`: Used to control the number of threads used when unzipping and installing
packages. See the [rayon documentation](https://docs.rs/rayon/latest/rayon/) for more.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this no longer have any effect? (If it still has an effect, we should leave it here, even if UV_CONCURRENT_INSTALLS is recommended, since this is intended to document env variables that could affect behavior.)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't have any effect because we're initialize the global thread pool manually.

- `MACOSX_DEPLOYMENT_TARGET`: Used with `--python-platform macos` and related variants to set the
deployment target (i.e., the minimum supported macOS version). Defaults to `12.0`, the
least-recent non-EOL macOS version at time of writing.
Expand Down
11 changes: 8 additions & 3 deletions crates/uv-configuration/src/concurrency.rs
Original file line number Diff line number Diff line change
Expand Up @@ -11,13 +11,18 @@ pub struct Concurrency {
///
/// Note this value must be non-zero.
pub builds: usize,
/// The maximum number of concurrent installs.
///
/// Note this value must be non-zero.
pub installs: usize,
}

impl Default for Concurrency {
fn default() -> Self {
Concurrency {
downloads: Concurrency::DEFAULT_DOWNLOADS,
builds: Concurrency::default_builds(),
builds: Concurrency::threads(),
installs: Concurrency::threads(),
}
}
}
Expand All @@ -26,8 +31,8 @@ impl Concurrency {
// The default concurrent downloads limit.
pub const DEFAULT_DOWNLOADS: usize = 50;

// The default concurrent builds limit.
pub fn default_builds() -> usize {
// The default concurrent builds and install limit.
pub fn threads() -> usize {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just confirming that this is the same as the Rayon default?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

std::thread::available_parallelism()
.map(NonZeroUsize::get)
.unwrap_or(1)
Expand Down
1 change: 1 addition & 0 deletions crates/uv-workspace/src/combine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ impl Combine for PipOptions {
.concurrent_downloads
.combine(other.concurrent_downloads),
concurrent_builds: self.concurrent_builds.combine(other.concurrent_builds),
concurrent_installs: self.concurrent_installs.combine(other.concurrent_installs),
}
}
}
Expand Down
1 change: 1 addition & 0 deletions crates/uv-workspace/src/settings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,5 @@ pub struct PipOptions {
pub require_hashes: Option<bool>,
pub concurrent_downloads: Option<NonZeroUsize>,
pub concurrent_builds: Option<NonZeroUsize>,
pub concurrent_installs: Option<NonZeroUsize>,
}
1 change: 1 addition & 0 deletions crates/uv/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ indicatif = { workspace = true }
itertools = { workspace = true }
miette = { workspace = true, features = ["fancy"] }
owo-colors = { workspace = true }
rayon = { workspace = true }
regex = { workspace = true }
rustc-hash = { workspace = true }
serde = { workspace = true }
Expand Down
12 changes: 12 additions & 0 deletions crates/uv/src/main.rs
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,10 @@ async fn run() -> Result<ExitStatus> {

// Resolve the settings from the command-line arguments and workspace configuration.
let args = PipCompileSettings::resolve(args, workspace);
rayon::ThreadPoolBuilder::new()
.num_threads(args.shared.concurrency.installs)
.build_global()
.expect("failed to initialize global rayon pool");

// Initialize the cache.
let cache = cache.init()?.with_refresh(args.refresh);
Expand Down Expand Up @@ -247,6 +251,10 @@ async fn run() -> Result<ExitStatus> {

// Resolve the settings from the command-line arguments and workspace configuration.
let args = PipSyncSettings::resolve(args, workspace);
rayon::ThreadPoolBuilder::new()
.num_threads(args.shared.concurrency.installs)
.build_global()
.expect("failed to initialize global rayon pool");

// Initialize the cache.
let cache = cache.init()?.with_refresh(args.refresh);
Expand Down Expand Up @@ -293,6 +301,10 @@ async fn run() -> Result<ExitStatus> {

// Resolve the settings from the command-line arguments and workspace configuration.
let args = PipInstallSettings::resolve(args, workspace);
rayon::ThreadPoolBuilder::new()
.num_threads(args.shared.concurrency.installs)
.build_global()
.expect("failed to initialize global rayon pool");

// Initialize the cache.
let cache = cache.init()?.with_refresh(args.refresh);
Expand Down
18 changes: 16 additions & 2 deletions crates/uv/src/settings.rs
Original file line number Diff line number Diff line change
Expand Up @@ -307,6 +307,7 @@ impl PipCompileSettings {
link_mode,
concurrent_builds: env(env::CONCURRENT_BUILDS),
concurrent_downloads: env(env::CONCURRENT_DOWNLOADS),
concurrent_installs: env(env::CONCURRENT_INSTALLS),
..PipOptions::default()
},
workspace,
Expand Down Expand Up @@ -415,6 +416,7 @@ impl PipSyncSettings {
require_hashes: flag(require_hashes, no_require_hashes),
concurrent_builds: env(env::CONCURRENT_BUILDS),
concurrent_downloads: env(env::CONCURRENT_DOWNLOADS),
concurrent_installs: env(env::CONCURRENT_INSTALLS),
..PipOptions::default()
},
workspace,
Expand Down Expand Up @@ -567,6 +569,7 @@ impl PipInstallSettings {
require_hashes: flag(require_hashes, no_require_hashes),
concurrent_builds: env(env::CONCURRENT_BUILDS),
concurrent_downloads: env(env::CONCURRENT_DOWNLOADS),
concurrent_installs: env(env::CONCURRENT_INSTALLS),
..PipOptions::default()
},
workspace,
Expand Down Expand Up @@ -955,6 +958,7 @@ impl PipSharedSettings {
require_hashes,
concurrent_builds,
concurrent_downloads,
concurrent_installs,
} = workspace
.and_then(|workspace| workspace.options.pip)
.unwrap_or_default();
Expand Down Expand Up @@ -1044,11 +1048,18 @@ impl PipSharedSettings {
downloads: args
.concurrent_downloads
.or(concurrent_downloads)
.map_or(Concurrency::DEFAULT_DOWNLOADS, NonZeroUsize::get),
.map(NonZeroUsize::get)
.unwrap_or(Concurrency::DEFAULT_DOWNLOADS),
builds: args
.concurrent_builds
.or(concurrent_builds)
.map_or_else(Concurrency::default_builds, NonZeroUsize::get),
.map(NonZeroUsize::get)
.unwrap_or_else(Concurrency::threads),
installs: args
.concurrent_installs
.or(concurrent_installs)
.map(NonZeroUsize::get)
.unwrap_or_else(Concurrency::threads),
},
}
}
Expand All @@ -1061,6 +1072,9 @@ mod env {

pub(super) const CONCURRENT_BUILDS: (&str, &str) =
("UV_CONCURRENT_BUILDS", "a non-zero integer");

pub(super) const CONCURRENT_INSTALLS: (&str, &str) =
("UV_CONCURRENT_INSTALLS", "a non-zero integer");
}

/// Attempt to load and parse an environment variable with the given name.
Expand Down
8 changes: 8 additions & 0 deletions uv.schema.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading