Inconsistent behavior with override-dependencies across package sources

[philipqueen]

I'm trying to use the overrides feature to exclude a package download, as described in #4422 (comment)

So, I added the following to my pyproject.toml:

[tool.uv]
override-dependencies = ["opencv-python; sys_platform == 'never'"]

The issue I'm trying to solve is that we depend on multiple pose estimation libraries, including mediapipe, which depends on opencv-python, and ultralytics, which depends on opencv-contrib-python, a superset of opencv-python. opencv-python and opencv-contrib-python conflict with each other, but running pip check with both in the same environment passes.

I tried testing the change with different package sources (based on this https://docs.astral.sh/uv/pip/packages/), and got different results based on different sources:

With uv pip install -e.: when installing from within the project folder, this works as intended - no opencv-python, the override works
With uv pip install 'freemocap @ /Users/philipqueen/Documents/GitHub/freemocap/': when installing from disk (with the correct branch active), this doesn't work. opencv-python is still downloaded, override doesn't seem to work
With uv pip install "git+https://github.com/freemocap/freemocap@philip/use_uv": when installing from git (on the correct branch), this doesn't work. opencv-python is still downloaded, override doesn't seem to work.

I may be doing something wrong, or using the feature outside its intended purpose, but it seems like a bug that it works with some package sources and not others.

My goal is that when I publish to PyPI, running uv pip install freemocap doesn't install opencv-python. I really appreciate uv adding support for this reature

Platform/Version Info

uv version: uv 0.5.1 (95e7d87 2024-11-12)
uv platform: mac 12.6.2

[charliermarsh]

In the pip interface, we don't respect the constraints in the pyproject.toml within an arbitrary project (which is how uv pip install sees freemocap @ /Users/philipqueen/Documents/GitHub/freemocap/. We only discover the constraints from the discovered configuration file, which would be the pyproject.toml in the current working directory or parent directory.

[charliermarsh]

It's not really intended to be part of the published package metadata -- you could view it as local project configuration.

[philipqueen]

Thanks for the quick response!

That makes sense as a general approach. Is there any way it could be extended to be part of the published package metadata?

The project I'm working on is designed for direct use by end users. Like a lot of end-user python packages, it's recommended the user creates an environment, and installs just our package in it. It would be great if there were some way to say "when you're installing freemocap, don't install opencv-python". If the user already has the conflicting package in their environment, that would be their fault, but now there doesn't seem to be a way around downloading the conflicting packages, uninstalling both, and reinstalling both. This is a big hurdle for non technical users.

[notatallshaw-gts]

That makes sense as a general approach. Is there any way it could be extended to be part of the published package metadata?

Package metadata is part of the Python packaging standards (https://packaging.python.org/en/latest/specifications/core-metadata/), such discussion to extend the metadata is held at: https://discuss.python.org/c/packaging/14

I'm not sure if uv could add it's own non-standard metadata, I know PyPI is now doing some metadata validation but not sure if it blocks unknown fields, but what would happen is if these packages got uploaded then uv would start treating packages differently than other installers (pip, poetry, pdm, hatch, etc.) and it would bifurcate the packaging ecosystem.

Though, expect to receive serious pushback around standardizing fields which allow for broken resolutions to be installed. Also be prepared to answer questions like, what if two different packages try to override each other, which one wins?

[notatallshaw-gts]

The project I'm working on is designed for direct use by end users. Like a lot of end-user python packages, it's recommended the user creates an environment, and installs just our package in it. It would be great if there were some way to say "when you're installing freemocap, don't install opencv-python". If the user already has the conflicting package in their environment, that would be their fault, but now there doesn't seem to be a way around downloading the conflicting packages, uninstalling both, and reinstalling both. This is a big hurdle for non technical users.

Your problem is not unique, but I don't know a "best way" to handle this, especially for non-technical users. Here are solutions I have seen:

• Create scripts that a user "just runs" that installs everything correctly inside an environment: https://github.com/lllyasviel/stable-diffusion-webui-forge?tab=readme-ov-file#installing-forge
• Provide a constraints (or lock) file that specifies everything exactly and instruct your users on how to use: https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installation-tools
• Have your requirements lock to specific versions: https://community.plotly.com/t/dash-is-planning-to-lock-flask-werkzeug-versions/74829
• Provide a single binary executable of your project (there are many different libraries that do this with different trade offs, but the most popular is https://pyinstaller.org/en/stable/ and then perhaps https://nuitka.net/)

There is definitely space for someone, like astral / uv 😉, to provide a disruptive solution that better answers how to distribute application-like Python tools to users. But I don't exactly know what the solution would look like.

[philipqueen]

@notatallshaw-gts Thanks for the thoughtful answer! It's definitely helping me understand python packaging better. I'll have to dig into the process more to understand fully.

As far as extending project metadata goes, I don't know if that's the best solution, or if what I want could be achieved other ways. As far as broken resolutions go, it's an interesting problem. The only working version of our software fails pip check (or uv pip check), and the versions that pass the check don't run. Of course, you could say this is ultimately a problem with opencv's python packaging, but they haven't responded to issues about it yet. So we're stuck working around a major player in the ecosystem.

If it's true the problem we're facing is deeper than Python can currently handle, it may be best to go the curl {installation_script} | sh route. I've never liked the security risk of that approach - it's easy enough for me to trust that the Rust Foundation is handling the security here well, much harder for me to trust myself to make such a script and avoid all the security pitfalls.

We've also tried the executable methods, and while the binary installers have failed, PyApp has been very promising. It's not perfect yet, but it does get some of our users over the technical hump.

Thanks again for the help. I think this is a real trouble in the python packaging ecosystem, and it would be amazing if the astral folks could find a way to handle it gracefully.