From 865e9e062626fa07e4c76d03be5fb395c5bd03c4 Mon Sep 17 00:00:00 2001 From: Charlie Marsh Date: Mon, 19 Aug 2024 14:57:31 -0400 Subject: [PATCH] Make some edits to the workspace concept documentation (#6223) --- docs/concepts/workspaces.md | 245 ++++++++++++++++++++++++++++-------- 1 file changed, 190 insertions(+), 55 deletions(-) diff --git a/docs/concepts/workspaces.md b/docs/concepts/workspaces.md index 12068a8b9395..988bdcd8ac8d 100644 --- a/docs/concepts/workspaces.md +++ b/docs/concepts/workspaces.md @@ -1,39 +1,32 @@ # Workspaces -Workspaces help organize large codebases by splitting them into multiple packages with independent -dependencies. Each package in a workspace has its own `pyproject.toml`, but they are all locked -together in a shared lockfile and installed to shared virtual environment. +Inspired by the [Cargo](https://doc.rust-lang.org/cargo/reference/workspaces.html) concept of the +same name, a workspace is "a collection of one or more packages, called _workspace members_, that +are managed together." -Using the project interface, `uv run` and `uv sync` will install all packages of the workspace, -unless you select a single workspace member with `--package`. When using the `uv pip` interface, -workspace dependencies behave like editable path dependencies. +Workspaces organize large codebases by splitting them into multiple packages with common +dependencies. Think: a FastAPI-based web application, alongside a series of libraries that are +versioned and maintained as separate Python packages, all in the same Git repository. -## When (not) to use workspaces - -One common use case for a workspace is that the codebase grows large, and eventually you want some -modules to become independent packages with their own dependency specification. Other use cases are -separating parts of the codebase with different responsibilities, e.g. in a repository with a -library package and CLI package, where the CLI package makes features of the library available but -has additional dependencies, a webserver with a backend and an ingestion package, or a library that -has a performance-critical subroutine implemented in a native language. +In a workspace, each package defines its own `pyproject.toml`, but the workspace shares a single +lockfile, ensuring that the workspace operates with a consistent set of dependencies. -Workspaces are not suited when you don't want to install all members together, members have -conflicting requirements, or you simply want individual virtual environments per project. In this -case, use regular (editable) relative path dependencies. +As such, `uv lock` operates on the entire workspace at once, while `uv run` and `uv sync` operate on +the workspace root by default, though both accept a `--package` argument, allowing you to run a +command in a particular workspace member from any workspace directory. -Currently, workspace don't properly support different members having different `requires-python` -values, we apply the highest of all `requires-python` lower bounds to the entire workspace. You need -to use a `uv pip` to install individual member in an older virtual environment. +## Getting started -!!! note +To create a workspace, add a `tool.uv.workspace` table to a `pyproject.toml`, which will implicitly +create a workspace rooted at that package. - As Python does not provide dependency isolation, uv can't ensure that a package uses only the dependencies it has declared, and not also imports a package that was installed for another dependency. For workspaces specifically, uv can't ensure that packages don't import dependencies declared by another workspace member. +!!! tip -## Usage + By default, running `uv init` inside an existing package will add the newly created member to the workspace, creating a `tool.uv.workspace` table in the workspace root if it doesn't already exist. -A workspace can be created by adding a `tool.uv.workspace` table to a `pyproject.toml` that will -become the workspace root. This table contains `members` (mandatory) and `exclude` (optional), with -lists of globs of directories: +In defining a workspace, you must specify the `members` (required) and `exclude` (optional) keys, +which direct the workspace to include or exclude specific directories as members respectively, and +accept lists of globs: ```toml title="pyproject.toml" [tool.uv.workspace] @@ -41,35 +34,140 @@ members = ["packages/*", "examples/*"] exclude = ["example/excluded_example"] ``` -`uv.lock` and `.venv` for the entire workspace are created next to this `pyproject.toml`. All -members need to be in directories below it. +In this example, the workspace includes all packages in the `packages` directory and all examples in +the `examples` directory, with the exception of the `example/excluded_example` directory. + +Every directory included by the `members` globs (and not excluded by the `exclude` globs) must +contain a `pyproject.toml` file; in other words, every member must be a valid Python package, or +workspace discovery will raise an error. + +## Workspace roots + +Every workspace needs a workspace root, which can either be explicit or "virtual". + +An explicit root is a directory that is itself a valid Python package, and thus a valid workspace +member, as in: + +```toml title="pyproject.toml" +[project] +name = "albatross" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = ["bird-feeder", "tqdm>=4,<5"] + +[tool.uv.sources] +bird-feeder = { workspace = true } + +[tool.uv.workspace] +members = ["packages/*"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" +``` + +A virtual root is a directory that is _not_ a valid Python package, but contains a `pyproject.toml` +with a `tool.uv.workspace` table. In other words, the `pyproject.toml` exists to define the +workspace, but does not itself define a package, as in: + +```toml title="pyproject.toml" +[tool.uv.workspace] +members = ["packages/*"] +``` + +A virtual root _must not_ contain a `[project]` table, as the inclusion of a `[project]` table +implies the directory is a package, and thus an explicit root. As such, virtual roots cannot define +their own dependencies; however, they _can_ define development dependencies as in: + +```toml title="pyproject.toml" +[tool.uv.workspace] +members = ["packages/*"] + +[tool.uv] +dev-dependencies = ["ruff==0.5.0"] +``` + +By default, `uv run` and `uv sync` operates on the workspace root, if it's explicit. For example, in +the above example, `uv run` and `uv run --package albatross` would be equivalent. For virtual +workspaces, `uv run` and `uv sync` instead sync all workspace members, since the root is not a +member itself. + +## Workspace sources + +Within a workspace, dependencies on workspace members are facilitated via +[`tool.uv.sources`](./dependencies.md), as in: + +```toml title="pyproject.toml" +[project] +name = "albatross" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = ["bird-feeder", "tqdm>=4,<5"] + +[tool.uv.sources] +bird-feeder = { workspace = true } + +[tool.uv.workspace] +members = ["packages/*"] + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" +``` + +In this example, the `albatross` package depends on the `bird-feeder` package, which is a member of +the workspace. The `workspace = true` key-value pair in the `tool.uv.sources` table indicates the +`bird-feeder` dependency should be provided by the workspace, rather than fetched from PyPI or +another registry. + +Any `tool.uv.sources` definitions in the workspace root apply to all members, unless overridden in +the `tool.uv.sources` of a specific member. For example, given the following `pyproject.toml`: + +```toml title="pyproject.toml" +[project] +name = "albatross" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = ["bird-feeder", "tqdm>=4,<5"] + +[tool.uv.sources] +bird-feeder = { workspace = true } +tqdm = { git = "https://github.com/tqdm/tqdm" } + +[tool.uv.workspace] +members = ["packages/*"] -If `tool.uv.sources` is defined in the workspace root, it applies to all members, unless overridden -in the `tool.uv.sources` of a specific member. +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" +``` -Using `uv init` inside a workspace will add the newly created package to `members`. +Every workspace member would, by default, install `tqdm` from GitHub, unless a specific member +overrides the `tqdm` entry in its own `tool.uv.sources` table. -## Common structures +## Workspace layouts -There a two main workspace structures: A **root package with helpers** and a **flat workspace**. +In general, there are two common layouts for workspaces, which map to the two kinds of workspace +roots: a **root package with helpers** (for explicit roots) and a **flat workspace** (for virtual +roots). -The root workspace layout defines one main package in the root of the repository, with helper -packages in `packages`. In this example `albatross/pyproject.toml` has both a `project` section and -a `tool.uv.workspace` section. +In the former case, the workspace includes an explicit workspace root, with peripheral packages or +libraries defined in `packages`. For example, here, `albatross` is an explicit workspace root, and +`bird-feeder` and `seeds` are workspace members: ```text albatross ├── packages -│ ├── provider_a +│ ├── bird-feeder │ │ ├── pyproject.toml │ │ └── src -│ │ └── provider_a +│ │ └── bird_feeder │ │ ├── __init__.py │ │ └── foo.py -│ └── provider_b +│ └── seeds │ ├── pyproject.toml │ └── src -│ └── provider_b +│ └── seeds │ ├── __init__.py │ └── bar.py ├── pyproject.toml @@ -80,9 +178,8 @@ albatross └── main.py ``` -In the flat layout, all packages are in the `packages` directory, and the root `pyproject.toml` -defines a so-called virtual workspace. In this example `albatross/pyproject.toml` has only a -`tool.uv.workspace` section, but no `project`. +In the latter case, _all_ members are located in the `packages` directory, and the root +`pyproject.toml` comprises a virtual root: ```text albatross @@ -93,16 +190,16 @@ albatross │ │ └── albatross │ │ ├── __init__.py │ │ └── foo.py -│ ├── provider_a +│ ├── bird-feeder │ │ ├── pyproject.toml │ │ └── src -│ │ └── provider_a +│ │ └── bird_feeder │ │ ├── __init__.py │ │ └── foo.py -│ └── provider_b +│ └── seeds │ ├── pyproject.toml │ └── src -│ └── provider_b +│ └── seeds │ ├── __init__.py │ └── bar.py ├── pyproject.toml @@ -110,15 +207,53 @@ albatross └── uv.lock ``` -In the flat layout, you may still define development dependencies in the workspace root -`pyproject.toml`: +## When (not) to use workspaces + +Workspaces are intended to facilitate the development of multiple interconnected packages within a +single repository. As a codebase grows in complexity, it can be helpful to split it into smaller, +composable packages, each with their own dependencies and version constraints. + +Workspaces help enforce isolation and separation of concerns. For example, in uv, we have separate +packages for the core library and the command-line interface, enabling us to test the core library +independently of the CLI, and vice versa. + +Other common use cases for workspaces include: + +- A library with a performance-critical subroutine implemented in an extension module (Rust, C++, + etc.). +- A library with a plugin system, where each plugin is a separate workspace package with a + dependency on the root. + +Workspaces are _not_ suited for cases in which members have conflicting requirements, or desire a +separate virtual environment for each member. In this case, path dependencies are often preferable. +For example, rather than grouping `albatross` and its members in a workspace, you can always define +each package as its own independent project, with inter-package dependencies defined as path +dependencies in `tool.uv.sources`: ```toml title="pyproject.toml" -[tool.uv.workspace] -members = ["packages/*"] +[project] +name = "albatross" +version = "0.1.0" +requires-python = ">=3.12" +dependencies = ["bird-feeder", "tqdm>=4,<5"] -[tool.uv] -dev-dependencies = [ - "pytest >=8.3.2,<9" -] +[tool.uv.sources] +bird-feeder = { path = "packages/bird-feeder" } + +[build-system] +requires = ["hatchling"] +build-backend = "hatchling.build" ``` + +This approach conveys many of the same benefits, but allows for more fine-grained control over +dependency resolution and virtual environment management (with the downside that `uv run --package` +is no longer available; instead, commands must be run from the relevant package directory). + +Finally, uv's workspaces enforce a single `requires-python` for the entire workspace, taking the +intersection of all members' `requires-python` values. If you need to support testing a given member +on a Python version that isn't supported by the rest of the workspace, you may need to use `uv pip` +to install that member in a separate virtual environment. + +!!! note + + As Python does not provide dependency isolation, uv can't ensure that a package uses its declared dependencies and nothing else. For workspaces specifically, uv can't ensure that packages don't import dependencies declared by another workspace member.