Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: pixi project export conda to export project to conda environment.yml's #1427

Closed
wants to merge 6 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
198 changes: 198 additions & 0 deletions src/cli/project/export/conda.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
use std::path::PathBuf;

use clap::Parser;

use itertools::Itertools;
use miette::IntoDiagnostic;
use rattler_conda_types::{MatchSpec, Platform};

use crate::cli::LockFileUsageArgs;
use crate::lock_file::UpdateLockFileOptions;
use crate::utils::conda_environment_file::{CondaEnvDep, CondaEnvFile};
use crate::{project, HasFeatures, Project};

// enum to select version spec formatting
#[derive(clap::ValueEnum, Clone, Debug)]
pub enum VersionSpec {
Manifest,
Locked,
None,
}

/// Exports a projects dependencies as an environment.yml
///
/// The environment is printed to standard out
#[derive(Debug, Parser)]
#[clap(arg_required_else_help = false)]
pub struct Args {
/// The platform to list packages for. Defaults to the current platform.
#[arg(long)]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[arg(long)]
#[arg(short, long)]

To align with other cli's.

pub platform: Option<Platform>,

/// The path to 'pixi.toml' or 'pyproject.toml'
#[arg(long)]
pub manifest_path: Option<PathBuf>,

/// The environment to list packages for. Defaults to the default environment.
#[arg(short, long)]
pub environment: Option<String>,

/// Name for environment
#[arg(short, long)]
pub name: Option<String>,

/// Dependency spec output method
#[arg(long, default_value = "manifest", value_enum)]
pub version_spec: VersionSpec,
Comment on lines +45 to +46
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[arg(long, default_value = "manifest", value_enum)]
pub version_spec: VersionSpec,
#[arg(long, default="false")]
pub lock: bool,

What do you think about this? to keep it simple.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I explained more of my thinking in https://github.com/prefix-dev/pixi/pull/1427/files#r1616283963 but I think there are really 3 different ways that versions are specified in environment.ymls, so the arg needs to be able to handle all three.

How about shortening version_spec to versions?pixi project export conda --versions locked I think that may convey the intent better than version_spec to more users.


#[clap(flatten)]
pub lock_file_usage: LockFileUsageArgs,

/// Don't install the environment for pypi solving, only update the lock-file if it can solve without installing.
#[arg(long)]
pub no_install: bool,
}

pub async fn execute(args: Args) -> miette::Result<()> {
let project = Project::load_or_else_discover(args.manifest_path.as_deref())?;
let environment = project.environment_from_name_or_env_var(args.environment)?;

let platform = args.platform.unwrap_or_else(|| environment.best_platform());

let name = match args.name {
Some(arg_name) => arg_name,
None => format!("{}-{}-{}", project.name(), environment.name(), platform),
};

let channels = environment
.channels()
.into_iter()
.map(|channel| channel.name().to_string())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
.map(|channel| channel.name().to_string())
.map(|channel| {
default_channel_config().canonical_name(channel.base_url()).to_string()
})

Otherwise the following channel would be exported as conda-forge:

channels = ["https://fast.prefix.dev/conda-forge"]

.collect_vec();

if let VersionSpec::Locked = args.version_spec {
let lock_file = project
.up_to_date_lock_file(UpdateLockFileOptions {
lock_file_usage: args.lock_file_usage.into(),
no_install: args.no_install,
..UpdateLockFileOptions::default()
})
.await?;

let locked_deps = lock_file
.lock_file
.environment(environment.name().as_str())
.and_then(|env| env.packages(platform).map(Vec::from_iter))
.unwrap_or_default();

let mut dependencies = locked_deps
.iter()
.filter_map(|d| d.as_conda())
.map(|d| CondaEnvDep::Conda(d.package_record().to_string()))
.collect_vec();

let mut pypi_dependencies = locked_deps
.iter()
.filter_map(|d| d.as_pypi())
.filter(|d| !d.is_editable())
.map(|d| format!("{}={}", d.data().package.name, d.data().package.version))
.collect_vec();

let editable_dependencies = environment
.pypi_dependencies(Some(platform))
.into_specs()
.filter_map(|(name, spec)| {
let requirement = spec
.as_pep508(name.as_normalized(), project.root())
.into_diagnostic()
.unwrap();
if let project::manifest::python::RequirementOrEditable::Editable(
_package_name,
requirements_txt,
) = &requirement
{
let relative_path = requirements_txt
.path
.as_path()
.strip_prefix(project.manifest_path().parent().unwrap());
return Some(format!("-e ./{}", relative_path.unwrap().to_string_lossy()));
}
None
})
.collect_vec();

pypi_dependencies.extend(editable_dependencies);

if !pypi_dependencies.is_empty() {
dependencies.push(CondaEnvDep::Pip {
pip: pypi_dependencies,
});
}

let env_file = CondaEnvFile {
name: Some(name),
channels,
dependencies,
};

let env_string = serde_yaml::to_string(&env_file).into_diagnostic()?;
println!("{}", env_string);

return Ok(());
}

let mut dependencies = environment
.dependencies(None, Some(platform))
.into_specs()
.map(|(name, spec)| match args.version_spec {
VersionSpec::Manifest => {
CondaEnvDep::Conda(MatchSpec::from_nameless(spec, Some(name)).to_string())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why would a user not want to add the given match spec?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm thinking about the range of ways that folks currently use environment.yml. I see a lot of environment.ymls passed around without any versions specified, and smaller numbers with just the direct dependencies constrained, and lesser again that have the fully resolved dependencies.

I'm trying to provide an option for those non version folks, by giving them an option (as well as to loosen constraints for various testing scenarios), while nudging them towards at least including the direct dependency match specs as default.

At the same time being able to support export the exact dependencies resolved, which is currently a separate code path, but I'd like to clean that up.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean, however this might not be the right place to introduce this loosing if dependencies logic. For instance there is another Issue talking about this here: #639.

If this would be a pixi wide solution which export would benefit from. I'd personally like to start with a match-spec or locked environment export mechanism. And later we could possibly add the --pin style here as well.

What do you think of that?

}
_ => CondaEnvDep::Conda(name.as_source().to_string()),
})
.collect_vec();

let pypi_dependencies = environment
.pypi_dependencies(Some(platform))
.into_specs()
.map(|(name, spec)| match args.version_spec {
VersionSpec::Manifest => {
let requirement = spec
.as_pep508(name.as_normalized(), project.root())
.into_diagnostic()
.unwrap();
return match &requirement {
project::manifest::python::RequirementOrEditable::Editable(
_package_name,
requirements_txt,
) => {
let relative_path = requirements_txt
.path
.as_path()
.strip_prefix(project.manifest_path().parent().unwrap());
format!("-e ./{}", relative_path.unwrap().to_string_lossy())
}
_ => requirement.to_string(),
};
}
_ => name.as_source().to_string(),
})
.collect_vec();

if !pypi_dependencies.is_empty() {
dependencies.push(CondaEnvDep::Pip {
pip: pypi_dependencies,
});
}

let env_file = CondaEnvFile {
name: Some(name),
channels,
dependencies,
};

let env_string = serde_yaml::to_string(&env_file).into_diagnostic()?;
println!("{}", env_string);

Ok(())
}
23 changes: 23 additions & 0 deletions src/cli/project/export/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
use clap::Parser;

mod conda;

#[derive(Debug, Parser)]
pub enum Command {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good that you prepared for future options!

Copy link
Contributor Author

@abkfenris abkfenris May 27, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not gonna promise that I'm gonna be the one to write all of them though!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I dare you 😝

#[clap(alias = "c")]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#[clap(alias = "c")]
#[clap(visible_alias = "c")]

Conda(conda::Args),
}

/// Commands for exporting dependencies to additional formats
#[derive(Debug, Parser)]
pub struct Args {
#[command(subcommand)]
command: Command,
}

pub async fn execute(cmd: Args) -> miette::Result<()> {
match cmd.command {
Command::Conda(args) => conda::execute(args).await?,
};
Ok(())
}
3 changes: 3 additions & 0 deletions src/cli/project/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -3,13 +3,15 @@ use std::path::PathBuf;

pub mod channel;
pub mod description;
pub mod export;
pub mod platform;
pub mod version;

#[derive(Debug, Parser)]
pub enum Command {
Channel(channel::Args),
Description(description::Args),
Export(export::Args),
Platform(platform::Args),
Version(version::Args),
}
Expand All @@ -28,6 +30,7 @@ pub async fn execute(cmd: Args) -> miette::Result<()> {
match cmd.command {
Command::Channel(args) => channel::execute(args).await?,
Command::Description(args) => description::execute(args).await?,
Command::Export(cmd) => export::execute(cmd).await?,
Command::Platform(args) => platform::execute(args).await?,
Command::Version(args) => version::execute(args).await?,
};
Expand Down
12 changes: 6 additions & 6 deletions src/utils/conda_environment_file.rs
Original file line number Diff line number Diff line change
Expand Up @@ -2,22 +2,22 @@ use itertools::Itertools;
use miette::IntoDiagnostic;
use rattler_conda_types::ParseStrictness::Lenient;
use rattler_conda_types::{Channel, MatchSpec};
use serde::Deserialize;
use serde::{Deserialize, Serialize};
use std::str::FromStr;
use std::{io::BufRead, path::Path, sync::Arc};

use crate::config::Config;

#[derive(Deserialize, Debug, Clone)]
#[derive(Serialize, Deserialize, Debug, Clone)]
pub struct CondaEnvFile {
#[serde(default)]
name: Option<String>,
pub name: Option<String>,
#[serde(default)]
channels: Vec<String>,
dependencies: Vec<CondaEnvDep>,
pub channels: Vec<String>,
pub dependencies: Vec<CondaEnvDep>,
}

#[derive(Deserialize, Debug, Clone)]
#[derive(Serialize, Deserialize, Debug, Clone)]
#[serde(untagged)]
pub enum CondaEnvDep {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think if you implement a impl From<MatchSpec> for CondaEnvDep it should be pretty easy to add them.

You already have the PacakgeName and NamelessMatchSpec in the manifest.

So it should be easy to create a MatchSpec and use the .to_string() to get the actual string. e.g.:

let spec = MatchSpec::from_nameless(nameless_spec, Some(package_name));

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm able to quickly get some of the specs to render ok with CondaEnvDep::Conda(format!("{}{}", name.as_source(), spec)), but a handful (*) aren't looking right.

For pixi's pixi.toml

name: pixi-default-osx-arm64
channels:
- conda-forge
dependencies:
- pre-commit~=3.3.0
- rust~=1.77.0
- openssl3.*
- pkg-config0.29.*
- git2.42.0.*
- cffconvert>=2.0.0,<2.1
- tbump>=6.9.0,<6.10

I haven't tried writing an impl yet.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It needs a space. But this is indeed basically it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without an impl that gets me closer CondaEnvDep::Conda(MatchSpec::from_nameless(spec, Some(name)).to_string()):

name: pixi-default-osx-arm64
channels:
- conda-forge
dependencies:
- pre-commit ~=3.3.0
- rust ~=1.77.0
- openssl 3.*
- pkg-config 0.29.*
- git 2.42.0.*
- cffconvert >=2.0.0,<2.1
- tbump >=6.9.0,<6.10

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wait, if there is a space and no leading operator, does that get interpreted as =?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes I believe so

Conda(String),
Expand Down
Loading