Locking of build dependencies

[sbidoul]

Coming from #3611 (comment) and following comments.

Caveat: this is certainly not a complete proposal, and I reckon I have not investigated the capabilities of uv lock/sync in any detail, so consider this as thoughts to start the conversation on the usefulness of such features.

What

The uv.lock format supports locking source distributions (from sdists, git source trees, ...). This is great!

I would find it useful that

• commands that install from a lock file fail (by default?) if they need to install build dependencies that are not locked
• the lock command grows a mean to lock the build system requirements of the source dependencies declared in the lock file it produces

Why

When installing from a lock file I expect that everything that the installation downloads (or obtains from cache) is checked against the hashes (or immutable VCS commit hashes) that are present in the lock file, for repeatability and security.

If, during installation, other artifacts are downloaded (or obtained from cache) without my knowledge (to build source distributions), the security and repeatability of the process is weakened.

Some thoughts

• this may have performance implications for the lock command if sdists have to be downloaded to know the build requirements (universal-lock: clean up lock file format #3611 (comment))
• some build dependencies are dynamic (get_requires_for_build_{wheel,editable}), further complicating the locking operation
• different locked source distributions may require different versions of the same build dependency
• a first valuable step could be to lock build dependencies for which no wheel is available, and warn when an artifact that is not locked is downloaded when syncing

[paveldikov]

I was going to raise this same issue as well. Currently build-system.requires is completely unconstrained/unconstrainable and this is causing problems with build reproducibility.

This is causing real-life inconveniences in our organisation as, for example, new versions of setuptools appear but are not yet given the all-clear by our scanning suite. Versions that are not cleared yield a HTTP 403 upon download. The end result is a de-facto intractable state where no one can build (or even editably install) packages. (unless the developer decides to hard-constrain @ pyproject.toml, which is terrible UX and also doesn't help with the dependency-on-sdist problem.)

(I will probably also raise a separate issue w.r.t the handling of 403 codes -- this is a pain point with pip and it would be awesome if uv could do better)

[charliermarsh]

Definitely agree that we should do this.

[charliermarsh]

I really want to do this (lock the build requirements).

[vlad-ivanov-name]

This is becoming more relevant with newer python versions that don't ship setuptools by default anymore

[charliermarsh]

I really want to do this and would like to spend time on it personally, but I'm worried about how it will intersect with PEP 751.

[vlad-ivanov-name]

Hmm but that's more about lockfiles in general, not specific to build-time deps?

Correct me if I'm wrong but to me this PEP looks to be in early stages, and uv.lock looks way more mature as it is now. I would expect the introduction of this PEP to be orthogonal to the goal of being able to lock build dependencies - in the end if uv does decide to adopt the format, the build dependency group will probably be locked in a similar way to other dependency groups regardless of the final lock file format

[notatallshaw]

I really want to do this and would like to spend time on it personally, but I'm worried about how it will intersect with PEP 751.

Last I saw build time dependencies were dropped from PEP 751: https://discuss.python.org/t/pep-751-lock-files-again/59173/222 (end of that post).

I don't think anything changed: https://peps.python.org/pep-0751/#locking-build-requirements-for-sdists (FYI I really hope the "it confused enough people " isn't directed at me, I had some clarifying questions, and was mostly happy with the answers).

[charliermarsh]

@notatallshaw -- No, I agree with you, it's just that: if we add support for it here, what happens if PEP 751 comes out and we can't translate that support to the PEP?

[bluss]

Could you use the space for tool specific data in the PEP 751 lock file? Or even using one of the main fields like groups using some kind of convention that makes it possible for locked build deps to co-exist with other named things stored there.

[notatallshaw]

Yeah, makes sense to see the finalized version of PEP 751 first.

[vlad-ivanov-name]

what happens if PEP 751 comes out and we can't translate that support to the PEP?

I mean, this wouldn't be the first feature that uv supports that vanilla python tooling doesn't. This is important enough to implement regardless of what python itself does; after all, even locking of regular dependencies is not really supported out of the box, doesn't mean that uv should be restricted by that

[dimbleby]

similar proposals in poetry, pdm: python-poetry/poetry#8261 (and several duplicates), pdm-project/pdm#2465

I mostly think that people who care about reproducible builds would be better served by avoiding sdists altogether with --only-binary or similar.