pants generate-lockfiles to use uv for dependency resolution

[tomecki]

Is your feature request related to a problem? Please describe.
As of now the pip python dependency resolution using pants generate-lockfiles can take a very long time. In order to improve the overall UX of pants the alternative dependency resolvers can be made available to the users.

Describe the solution you'd like
https://github.com/astral-sh/uv is designed to be pip-compatible package installer and resolver. From my experience the uv pip compile is orders of magnitude faster than default pip dependency resolution.

Describe alternatives you've considered
This feature could be implemented as opt-in, given the maturity stage of the uv project.

[benjyw]

Duplicate of pex-tool/pex#2371 - see the discussion there, and feel free to join in!

[jsirois]

I'm not so sure this is a duplicate. Pants need not use Pex lock files. It could use uv directly and only support what it supports (for now); i.e.only locking for the current interpreter, not supporting subsetting and not supporting packed layout. It may be worth the effort to re-understand all the crazy things Pex does to support shortcomings in Pants perf. Perhaps there have been sandboxing improvements to obviate the need for packed layout or even the need for subsetting. Perhaps using multiple platforms specific lock files instead of 1 universal lock file makes sense, etc.

[benjyw]

Reopening this to discuss the possibility of Pants using uv directly, outside of Pex.

[brendan-morin]

Cross-posting my comment to the above linked issue here:

Chiming in: After seeing uv announced (and the enormous hype) it's clear the Astral team has quite effectively put a finger on the pulse of the python community and has created momentum and excitement in a way we don't often see.

It's too early to understand exactly where things will end up, but there is a decent chance that if uv executes on their vision, it has a legitimate chance to become ubiquitous as a third party package management tool in a similar way that e.g. requests become the defacto standard for python http interface.

In light on this, I think it would make sense for pants to lean into the zeitgeist, and embrace seamless uv integration as a core feature.

[sureshjoshi]

It's too early to understand exactly where things will end up, but there is a decent chance that if uv executes on their vision, it has a legitimate chance to become ubiquitous as a third party package management tool in a similar way that e.g. requests become the defacto standard for python http interface.

Or Python becomes Javascripty with an increasingly fragmented set of tools which all purport to solve the same suite of problems, claiming to be faster/est, while not supporting 100% compatibility with the solution they want to dethrone.

Oh how I sincerely hope that's not the case.

Don't get me wrong, we've got a lot of tools to solve the same set of problems, but it's not quite JS levels yet.

[jasondamour]

uv lock with multi-platform lockfiles shipped yesterday. I'm going to play around with it in pants

[jasondamour]

Jeez it feels so ridiculous to write dependencies back to a temporary pyproject.toml for uv to read. It'd be nice to directly invoke uv with a list of deps. Alternatively, it would be so much easier to take the stance that pants only enables uv if dependencies are in pyproject.toml (ruling out python_requirement())

[benjyw]

I thought uv was designed to be a drop-in replacement for pip, and as such surely can read requirements.txt files too?

Where are you dropping uv in though? As a replacement for pex for lockfile generation? How will that lockfile then be consumed?

The alternative is to try and integrate uv as an optional replacement for pip inside pex, as discussed here.

[jonasrauber]

uv can read requirements.txt. We use it as a first-pass resolver to check if there is a solution and to quickly iterate until there is. When we know there is solution, then we run pants/pex to generate the actual lockfile (independent of uv).

This works well today, without changes to pants, and saves us a lot of time / makes resolving actually feasible.

Alternatively, it would probably also be easily possible to just feed the output of uv as “requirements.txt” to pex and generate a compatible lockfile that way. That should be fast because everything is already resolved and pinned exactly by uv. But haven’t tried this variant yet. (To a small extent I do this by taking a few results from uv’s output as constraints.txt for pex in the above approach, to speed it up.)

This should work today, also without changes to pants.

[benjyw]

TBH I'm still unclear on where the performance benefits of uv come from. If dists are available as wheels then downloading them is network-bound and the implementation shouldn't matter. If you must build an sdist then that requires invoking Python. So is it that the implementation of the resolver algorithm is much faster and this turns out to be the bottleneck in practice?

[jonasrauber]

I was wondering the same. Haven’t digged into it yet. But it’s day and night. I think the big difference is when you have a deeply hidden conflict. That’s where pex takes an open ended amount of time (I give up before it finishes) whereas uv figures it out quickly. It looked a little bit like pex might just get stuck on some “bad” package versions so that it benefited from lower bounds. Maybe it also runs into throttling from the PyPI server (unlike uv which makes fewer requests when not considering multiplatform).

[benjyw]

Pex is delegating to Pip for the resolver algorithm, just to be clear.

So yeah, I guess it really is the resolver.

[StephanErb]

uv is fast because they do the same thing as pip's --use-feature=fast-deps and optimize for this aggressively: instead of downloading the full wheel, they use HTTP range requests to just download the wheel metadata from the server. Then the resolver can already run while the full wheels are downloading

Then then don't care too much about backwards compatibility of older wheel, egg formats, or even older Python versions.

The rest is then the faster resolver and aggressive caching.

[jasondamour]

I thought uv was designed to be a drop-in replacement for pip, and as such surely can read requirements.txt files too?

Where are you dropping uv in though? As a replacement for pex for lockfile generation? How will that lockfile then be consumed?

The alternative is to try and integrate uv as an optional replacement for pip inside pex, as discussed here.

@benjyw I was taking a stab at uv subsystem in pants for lockfile generation and consumption, and I was referencing how poetry locks worked in pants before pex which is why I mentioned writing to a tmp pyproject.toml:

[sureshjoshi]

Ran into this, which might have some light insights into practical uv perf benefits, and why:
https://blog.streamlit.io/python-pip-vs-astral-uv/

Is there a world where, instead of starting with uv, we'd try to plug and play some of the components they've built?

https://astral.sh/blog/uv

For those deep in the packaging ecosystem, uv also includes standards-compliant Rust implementations of PEP 440 (version identifiers), PEP 508 (dependency specifiers), PEP 517 (a build-system independent build frontend), PEP 405 (virtual environments), and more.