RFC: Tool management #3560
Comments
zanieb
added
the
tracking
|
As an aside, I'm a little uncertain about providing two interfaces i.e. |
|
A couple of things I noticed on reading:
|
|
|
Looks great! Two questions:
|
For consistency, I suggest using |
|
Thanks for the comments everyone, they're greatly appreciated.
The idea is that I agree that system level installations would need to be specified more, basically figuring out where to put things. I don't think this is particularly clear yet. On each platform, we need a list of candidate directories, filtered by locations it would not be surprising if we wrote to, that we have write access to, and are on the current
This is a great question and highlights how intertwined tool management and development dependencies are. The
This will definitely be supported by extending the entry for the tool in a Just to expand a bit on the difference here, we do not want to change the Presumably, as you mention, a shebang can point to the proper tool environment. I guess that's how globally installed executables will be implemented as well :) I hadn't thought much about how we want entry points to work yet, but yeah of course it's fine running an executable that needs a virtual environment that differs from the current one. I'll update the document accordingly. Thanks!
A very simple example is that the tool's dependencies conflicts with your project's dependencies. We might also not support multiple development dependency groups as in, e.g.,
There are a couple parts to this, but the short answer is: yes, we hope to solve this. I think there's a bit of lurking complexity in when symbolic links are resolved and pyenv shims — I'm not sure I can guarantee that we will resolve that specific issue, there are similar issues already, i.e., #1795. However, we do intend to perform toolchain management (i.e. install and manage Python for you) and we've invested a lot of time into proper discovery of existing Python interpreters. For both of those cases, we'll need to handle upgrades and switched Python versions. Ideally you'll be able to request a Python version for you tool and it'll just work. I appreciate that you highlighted this use-case though, we'll definitely need to be careful about interpreter resolution for tool environments. We won't be using |
|
|
Yeah I don't specify this command because I'm unsure if it's going to be necessary but we definitely care about this use case. For example, depending on how we implement entry points, we could just discover the latest interpreter automatically. We might also include a |
|
For Also, pipx assumes if there's a single console entry point, that that' what Also, |
This is mostly a shorter version of `uv run` that infers a requirement name from the command. The main goal here is to do the smallest amount of work necessary to get #3560 started. Closes #3613 e.g. ```shell $ uv tool run -- ruff check warning: `uv tool run` is experimental and may change without warning. Resolved 1 package in 34ms Installed 1 package in 2ms + ruff==0.4.4 error: Failed to parse example.py:1:5: Expected an expression example.py:1:5: E999 SyntaxError: Expected an expression Found 1 error. ```
|
I second @JelleZijlstra's concerns regarding the
I see your point here, and if this was already implemented I would agree. I may be missing something though, what are your thoughts? Excited to see this through! |
I really like this, it's really annoying when
If
Why can't uvx auto update the PATH for the user?
Would this also update the tools to use the latest version of python installed?
I'm not sure I fully understand this. Could you provide an example please?
For commands, I'd print a warning, and then try to run the
If I understand correctly, could
Could we try and then warn on failure?
Yes, if the dedicated command would un-install and then re-install, and installing on top hasn't seemed to work on all experiences
To what benefit?
My initial thoughts would be to re-install the tool. I can imagine a case where some tool needs non python packages installed and linked a certain way that a simple addition would not work for. Thanks for putting this up online, it's written quite well! |
|
@ekohilas thanks for your comments! (#3560 (comment))
When the cache entry is missing. Perhaps we'd allow a user to specify that they want to check for the latest version on every invocation but that seems like a rare use-case.
We cannot mutate the environment of a running shell. We can request that the user add it to their shell startup script though.
Unclear. It'd probably make sense unless the user requested a specific version of Python.
I don't think this is actually an open question anymore. If you do
I'm not sure I follow. The idea with
This seems weird. I think I'd upgrade all of the packages in the environment
Oof I did not realize Regarding the continued concerns about confusion w.r.t. |
|
Question -- will there be a place where tool dependencies are pinned? Normally transitive dependencies of dev-dependencies would be part of a lockfile, but if it's installed globally is there a risk of tool runs not being reproducible between instances of a project when the project references a tool? |
|
IMO, I'd recommend supporting the |
|
I believe in that case you'd want
|
|
Oh sorry, for install, you'd want |
What about this use case? |
Installing a package installs all of its entry points, that should just work? |
|
Ah, tried again and now it works 🤔 nm me. |
I am pretty hesitant to support this without a strong, motivating use-case. I don't think packages should be defining entry points for specific tools (even if other tools can read it). If there's a compelling story here, maybe there should be a PEP to canonicalize it? |
|
The tool stuff is very nice, but sort of following on from a previous question I had about using tools and the environments/interpreters they are run with, I am still a little unsure about a specific tool example, namely how pyinstaller ought to be run within the uv concept. (It possibly generalizes to a question of how the design intends tools to be run that rely on seeing/using the venv/dependencies of a project when run to do their job.) As it is a Python package used via a command-line interface, I would imagine pyinstaller is something that fits into the concept of a "tool". But to work,
(Something I think would be incredibly cool is if uv offered a |
|
I'd suggest Alternatively, you could do I think your use-case is pretty clearly addressed in the design though, which says:
It's not clear when we'll implement project-level tools though. |
Ah sorry, you're completely right, I moved over the project-level tools part a bit too fast. I agree that it's clear, consider my question void.
Thanks for the response, these are nice solutions, especially the nuance of the second. The only thing I would say is that they are a little non-obvious to someone as green as me as they aren't just a direct transfer of the "normal" way to run pyinstaller, so I hope others find this if they ever need it. Since the more general question doesn't make sense anyway, I don't know if you can take my question and your answer out of this thread and put it in its own (closed) issue to make it more findable? |
|
|
|
@edgarrmondragon feel free to make a targeted issue, yeah. |
Do you have stats to indicate that |
I have never used pipe run, but only because pipx puts tool alias into bin folder => there is no need to run But running the tool should definitely orders of magnitude more often that installing one... Based on the blog post, I figured uv is not creating a shebanged scripts to allow running tools as if they were binaries on the PATH...so it doesn't seem like a target use case as of today. |
No, but you run things many times and you typically only install them once :) I guess if you have a strong preference for the full interface,
We do put the shebanged scripts in the PATH as pipx does. |
|
I use pipx run all the time, and use pipx install once in a long while, usually to inject stuff. Which uv’s --with does. I like that it keeps everything up to date (never have to think about what is already installed or what needs updating, regardless of the machine I’m on). |
|
And, to be clear, since I think there’s some confusion above, they are alternatives of each other, you do not use run on a script you’ve installed (either pipx or uv). |
|
How much clarity it will bring if we rename # Add a tool to the project
uv global add ruff
# Add a tool with a specific version to the project
uv global add ruff==0.3.0
# Add a tool with an extra dependency
uv global add --with ruff-plugin ruff==0.3.0
# Add a tool to the project that requires the project dependencies at runtime
uv global add pytest --with-projectWhen I first read docs of rye I was confused about what tool does but I end up realizing it's just global dependency which we can run from anywhere. If my above understanding is correct then What do you guys think?
|
The problem is that you're not able to use that dependency as a Python library, just its CLI. That's why we've avoided terminology like
We don't have Discussions turned on in this repository and it didn't feel worth it for this one thing. Might be useful in the future. |
This is a tracking issue and design for management of "tools" in uv, similar to
pipx(see #1173).If this design does not fulfill your use-case, let us know in a comment describing what's missing. Please don't comment asking for status updates or a timeline — we'll be working on this actively and everything will be linked to this issue.
👋 Thanks for your interest in this design! I'll be editing this post to reflect the state of discussion, it will not be a static document
Roadmap
uv tool run#3613Overview
Tools are packages that provide executable entry points.
Tools may be installed, allowing their executable to be used without activation of an environment, e.g.
uv tool install ruffthenruffis available. Installation of a tool allows a developer to use the tool across all of their projects. Each tool is installed into an isolated Python environment that is abstracted from the user.Tools may be added to a project, allowing their executable to be used via
uv run,uv tool run, and detected by other tools such as IDEs. Adding a tool to a project allows developers to use a shared environment while working on a specific project. Adding a tool to a project differs from adding a development dependency in that the tool environment is isolated from the project’s environment by default. Tools in a project cannot depend on each other.The environment used to provide the tool is intended to be abstracted from the user. However, in some cases the user may need to install multiple packages to provide the tool they need e.g. a main package and a plugin. This must be done at declaration or install time, editing tool environments is not supported.
The following commands would be provided
uv tool run: Run a tool without installationuv tool install: Install a tool, allowing usage in a terminaluv tool uninstall: Remove an installed tooluv tool add: Add a tool to the current projectuv tool remove: Remove a tool from the current projectuv tool upgrade: Upgrade tools (either all or the specified tool)uv tool list: List available toolsuv tool show: Show details about a toolRunning tools
Tools can be run with
uv tool run. This command will choose the appropriate version of the tool to run based on the working directory. For example, if in a project that contains configuration for the tool, the project's version of the tool will be used. Otherwise, the tool installed at the user level will be used. If the tool is not in the project or installed, the latest cached version of the tool will be used. If the tool is not cached, the latest version will be added to the cache. The--isolatedflag can be used to opt-in to using the latest cached version instead of project or installed versions. The--latestflag can be used to force a check for a newer version (this differs from--no-cachein that the cache can still be used for the tool if it is the latest version and for the tool's dependencies). The user can opt-in to a specific version of the tool with<name>@<version>syntax, this will bypass different project or installed versions.A
uvxalias will be provided to run tools. Notably, this would be an alias foruv tool runnotuv tool— this could be a point of confusion for users transition frompipxbut avoids the need to typeuvx run. If we're providing an alias, it should be for the most-used command not for the broaderuv toolinterface. This matches tools in other ecosystems, likenpx.Additional dependencies may be requested when running a tool, e.g. with a
--with <spec>option. These will be installed in the tool's environment. If an installed tool's environment does not contain the request, a new, ephemeral environment will be created to fulfill the invocation — the installed tool's environment will not be modified. By default, the entry points provided by additional dependencies are not available. A flag may be provided to opt-in to this, e.g.pipxprovides a--include-depsoption.Unlike
uv run, the command name and requirement name are assumed to match e.g.uvx blackwill runblackwith an implicit requirement on the packageblack. Users may opt-out of this implicit requirement by providing a different package name e.g. withuvx --package foo -- bar.Examples:
Installing tools
Tools can be installed with
uv tool install. Tools can be installed into a user or system namespace, though the system namespace is more difficult to manage and should be considered a secondary objective during implementation. An installed tool is available on thePATHand usable from a terminal without activation of an environment. All entry points provided by the tool package are included by default. The user may opt to include a subset of entry points. If the entry points conflict with other managed tools at the same scope, installation should fail. If the entry points conflict with other executable names, a warning should be displayed during installation.Similar to
uv tool run, additional dependencies may be requested and the package may be manually specified.The requirement name is used as a "key" for identifying the tool in subsequent interactions e.g. uninstall and upgrade. This behavior may be confusing, if a user requests an operation on a command name that belongs to a requirement. In this case, a hint should be provided explaining the relationship.
When installing a tool, uv will create an entry in its configuration directory to track tool state. This entry is considered the source of truth for the tool's environment.
When installing a tool, a Python environment must be created for its package and dependencies. This environment should be entirely abstracted from the user. The environment should be created in a uv application data directory (platform dependent path).
When installing a tool, its entry points must be linked to executables in a directory on the user's
PATH. Multiple candidate directories may be considered with a preference for a directory present in thePATH. If none of the candidates are present in thePATH, the XDG standard directory should be used and a warning should be provided to the user prompting them to add it to thePATH. The user may provide an alternative installation path e.g.--root <path>(as incargo). This may be preferable to a--systemflag.As described in
uv tool run, by default, the entry points provided by additional dependencies are not installed. A flag may be provided to opt-in to installation of all entry points in the dependency tree, e.g.pipxprovides a--include-depsoption.Each entry point is added to the directory with its normal name, and
name@<version>. This suffix allows tools with conflicting versions to be used across different scopes. The user may customize this suffix with a--suffixflag. If a custom suffix is provided, we will not include the plain name executable during installation.When specifying suffixes, the
@is implied and will be included if the suffix begins with an alphanumeric character — if included by the user it will be trimmed for compatibility withpipx(which requires it to be included explicitly by the user). Leading characters such as-or_will drop the implied@allowing customization of suffixes. This behavior is not considered necessary for the initial implementation of this feature.uv does not allow installation of multiple tools in the same scope with the same key. If installation of multiple versions of a tool is desired, a suffix must be used. If the user attempts to install another version of an already-installed tool, the existing tool will be replaced. If the existing tool installation contains dependency requests that are not requested in the invocation that would replace it, an error should be raised instead.
Examples:
Tools in a project
Tools can be added to a project with
uv tool add. This is loosely an alternative to dependency groups — many uses of development dependency groups are for managing project-level tools. We may need to adjust this design depending on the status of PEP 735. This is likely the last part of the design we would implement, as it is the highest risk.This operation requires a
pyproject.tomlto be available.A user may opt-in to having their project’s dependencies available in the tool environment. However, tools may never share environments with each other.
Adding a tool to a project should:
.uv/tools/<key>.[tool.uv.tools]namespace in thepyproject.toml.binRunning
uv syncshould ensure that all project-level tools are available.Project-level tools are not added to the
PATHautomatically. Tool executables must be invoked withuv tool run,uv run, or in an activated environment.uv runshould detect and respect tool executables in the project. However, entry points provided by packages inuv runshould take precedence over tools. Placing tools into the virtual environment's bin should allow other tools like IDEs to discover the entry points.As described in
uv tool install, suffixes must be provided for tools if multiple versions are needed in the project.Examples:
Upgrading tools
Tools can be upgraded to newer version with
uv tool upgrade.The scope will be determined by the working directory. If in a project, this will upgrade the tools in the project. Otherwise, this will upgrade installed tools. The user may explicitly select a scope with a
--projectflag (which will fail if not in a project) or a--userflag. The--systemflag is required to upgrade system tools.If no tool name is provided, the user will be prompted to confirm an upgrade of all tools. If one or more tool names are provided, we will upgrade the given subset. If any of the given names are not found, we should throw an error.
The uv tool entry will be consulted to determine the requirement specification originally used for a tool. By default, will only be upgraded within the specification e.g. if the tool is pinned to a version, it will not be upgraded or if the tool has an upper bound it will not exceed it during upgrades. A flag may be provided to bump pinned packages, but the design of this behavior and its interaction with package specifications with upper bounds is outside the scope of this design and consequently deferred. In the interim, it may be useful to provide a command to view outdated tools or tools that would be upgraded if the restrictions were loosened.
Examples:
Open questions
uvx install?uv tool add?@?The text was updated successfully, but these errors were encountered: