pixi resolution order for composing features to make environments

[matthewfeickert]

Note

This Discussion exists as a GitHub Gist with pixi.toml and pixi.lock at: https://gist.github.com/matthewfeickert/9bcf0cc2ddfe27cd3786ab8531e67e9c

$ pixi --version
pixi 0.30.0

For testing purposes, I'm trying to create lock files for an ecosystem of tools across multiple Python versions.
To avoid having to repeat describing the dependencies, I'd like to use pixi's features to define the tools once in one feature (with lower bounds only as this would be part of nightly/weekly solves) and then have the Python version exist in a separate feature.
The goal here is to be able to create environments quickly by composing features together like

[environments]

tools = ["tools"]

tools-py3x = ["py3x", "tools"]

This works as long as everything in the tools feature is able to be resolved on conda-forge for the Python version in the corresponding Python feature.
However, one of the dependencies I'm using, coffea, seems to be misspecified on conda-forge under Python 3.8 (or maybe there is some other issue, but I need to follow up with the maintainers to check) and resolution seems to only be valid for coffea = ">=2024.1.0,<2024.3.0".
However, on PyPI, the resolution for coffea>=2024.1.0 works fine and so the valid environment can be created from fully specifying a single feature

[feature.tools-py38.dependencies]
python = "3.8.*"
...

[feature.tools-py38.pypi-dependencies]
# coffea on conda-forge seems to be misspecified for Python 3.8
# as only works for coffea = ">=2024.1.0,<2024.3.0"
coffea = ">=2024.1.0"
...

[environments]

tools-py38-standalone = ["tools-py38"]

However, trying to continue with composing features by reusing the tools feature and specifying coffea as a PyPI dependency in a py38 feature does not work, as there isn't any way to indicate that a feature should have a subtractive effect on another feature or that PyPI should be given priority. So the following does NOT work

[feature.py38.dependencies]
python = "3.8.*"

[feature.py38.pypi-dependencies]
# coffea on conda-forge seems to be misspecified for Python 3.8
# as only works for coffea = ">=2024.1.0,<2024.3.0"
coffea = ">=2024.1.0"

[environments]

tools = ["tools"]

tools-py38 = ["py38", "tools"]

Is there any way to compose features to make environments but have the ability to override or remove dependencies from one of the feature in the set? Or is the only way forward if there are problems with a package on the index (like here) to compose a bespoke feature?

Example pixi.toml for all use cases described:

[project]
authors = ["Matthew Feickert <matthew.feickert@cern.ch>"]
channels = ["conda-forge"]
description = "Add a short description here"
name = "pixi-resolution-order-example"
platforms = ["linux-64"]
version = "0.1.0"

[tasks]

[dependencies]

[feature.tools.dependencies]
# This feature is intentionally over-specified to be able to control
# the versions of subdependncies with more control.
# e.g. uproot is already a dependency of coffea
uproot = ">=5.0.0"
hist = ">=2.7.0"
mplhep = ">=0.3.50"
cabinetry = ">=0.6.0"
coffea = ">=2024.1.0"
func-adl = ">=3.3.0"
xrootd = ">=5.7.0"
dask-awkward = ">=2024.1.0"

[feature.tools.pypi-dependencies]
func-adl-servicex = ">=2.2"
servicex = ">=3.0.0"
fastjet = ">=3.4.0.0"  # fastjet on conda-forge is broken


[feature.py38.dependencies]
python = "3.8.*"

[feature.py38.pypi-dependencies]
# coffea on conda-forge seems to be misspecified for Python 3.8
# as only works for coffea = ">=2024.1.0,<2024.3.0"
coffea = ">=2024.1.0"

[feature.tools-py38.dependencies]
python = "3.8.*"
uproot = ">=5.0.0"
hist = ">=2.7.0"
mplhep = ">=0.3.50"
cabinetry = ">=0.6.0"
func-adl = ">=3.3.0"
xrootd = ">=5.7.0"
dask-awkward = ">=2024.1.0"

[feature.tools-py38.pypi-dependencies]
# coffea on conda-forge seems to be misspecified for Python 3.8
# as only works for coffea = ">=2024.1.0,<2024.3.0"
coffea = ">=2024.1.0"
func-adl-servicex = ">=2.2"
servicex = ">=3.0.0"
fastjet = ">=3.4.0.0"  # fastjet on conda-forge is broken

[feature.py311.dependencies]
python = "3.11.*"

[feature.py311.tasks]
export-env = """
mkdir -p tools/3.11 && \
pixi project export conda-environment --environment tools-py311 tools/3.11/environment.yml
"""

[environments]

tools = ["tools"]

# Composing features works as intended for tools-py311
tools-py311 = ["py311", "tools"]

# Composing features seems to not work for tools-py38,
# suggesting that
# tools-py38 = ["py38", "tools"]

# Fully recreating the feature does work though
tools-py38-standalone = ["tools-py38"]

[ruben-arts]

Thanks for the write-up @matthewfeickert, as far as I understand the question this is indeed not possible right now. Mixing conda and pypi dependencies currently works only one way. That being "use conda when you can". We've not yet been able to go back and forth between the resolution of the two solves.

I think a "marker" like the pypi world has with python_version < 3.9 in the version description would be really cool, as it might then be possible to setup these requirements in a different way. But that is currently only possible by manually splitting the features.

I understand this is not a helpful message but good to keep this idea in the back of our mind while we improve pixi.

[matthewfeickert]

Thanks @ruben-arts! This is helpful just to understand what is possible at the moment and if I'm missing anything. 👍