Skip to content

Commit

Permalink
Make some edits to the workspace concept documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
@charliermarsh
charliermarsh committed Aug 19, 2024
1 parent a4aef29 commit f583e8b
Showing 1 changed file with 188 additions and 56 deletions.
244 changes: 188 additions & 56 deletions docs/concepts/workspaces.md
View file
Original file line number Diff line number Diff line change
@@ -1,75 +1,170 @@
# Workspaces

Workspaces help organize large codebases by splitting them into multiple packages with independent
dependencies. Each package in a workspace has its own `pyproject.toml`, but they are all locked
together in a shared lockfile and installed to shared virtual environment.
Inspired by the [Cargo](https://doc.rust-lang.org/cargo/reference/workspaces.html) concept of the
same name, a workspace is "a collection of one or more packages, called _workspace members_, that
are managed together."

Using the project interface, `uv run` and `uv sync` will install all packages of the workspace,
unless you select a single workspace member with `--package`. When using the `uv pip` interface,
workspace dependencies behave like editable path dependencies.
Workspaces organize large codebases by splitting them into distinct packages with independent
dependencies. Think: a FastAPI-based web application, alongside a series of libraries that are
versioned and maintained as separate Python packages, all in the same Git repository.

## When (not) to use workspaces
In a workspace, each package defines its own `pyproject.toml`, but the workspace shares a single
lockfile, ensuring that the workspace operates with a consistent set of dependencies.

One common use case for a workspace is that the codebase grows large, and eventually you want some
modules to become independent packages with their own dependency specification. Other use cases are
separating parts of the codebase with different responsibilities, e.g. in a repository with a
library package and CLI package, where the CLI package makes features of the library available but
has additional dependencies, a webserver with a backend and an ingestion package, or a library that
has a performance-critical subroutine implemented in a native language.
As such, `uv lock` operates on the entire workspace at once, while `uv run` and `uv sync` operate on
the workspace root by default, though both accept a `--package` argument, allowing you to run a
command _in_ any workspace member _from_ any workspace directory.

Workspaces are not suited when you don't want to install all members together, members have
conflicting requirements, or you simply want individual virtual environments per project. In this
case, use regular (editable) relative path dependencies.
## Getting started

Currently, workspace don't properly support different members having different `requires-python`
values, we apply the highest of all `requires-python` lower bounds to the entire workspace. You need
to use a `uv pip` to install individual member in an older virtual environment.
To create a workspace, add a `tool.uv.workspace` table to a `pyproject.toml`, which will implicitly
create a workspace rooted at that package.

!!! note
In defining a workspace, you must specify the `members` (required) and `exclude` (optional) keys,
which direct the workspace to include or exclude specific directories as members respectively, and
accept lists of globs:

```toml title="pyproject.toml"
[tool.uv.workspace]
members = ["packages/*", "examples/*"]
exclude = ["example/excluded_example"]
```

In this example, the workspace includes all packages in the `packages` directory and all examples in
the `examples` directory, with the exception of the `example/excluded_example` directory.

As Python does not provide dependency isolation, uv can't ensure that a package uses only the dependencies it has declared, and not also imports a package that was installed for another dependency. For workspaces specifically, uv can't ensure that packages don't import dependencies declared by another workspace member.
Every directory included by the `members` globs (and not excluded by the `exclude` globs) must
contain a `pyproject.toml` file; in other words, every member must be a valid Python package.

## Usage
## Workspace roots

A workspace can be created by adding a `tool.uv.workspace` table to a `pyproject.toml` that will
become the workspace root. This table contains `members` (mandatory) and `exclude` (optional), with
lists of globs of directories:
Every workspace needs a workspace root, which can either be explicit or "virtual".

An explicit root is a directory that is itself a valid Python package, and thus a valid workspace
member, as in:

```toml title="pyproject.toml"
[project]
name = "albatross"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["bird-feeder", "tqdm>=4,<5"]

[tool.uv.sources]
bird-feeder = { workspace = true }

[tool.uv.workspace]
members = ["packages/*", "examples/*"]
exclude = ["example/excluded_example"]
members = ["packages/*"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

`uv.lock` and `.venv` for the entire workspace are created next to this `pyproject.toml`. All
members need to be in directories below it.
A virtual root is a directory that is _not_ a valid Python package, but contains a `pyproject.toml`
with a `tool.uv.workspace` table. In other words, the `pyproject.toml` exists to define the
workspace, but does not itself define a package, as in:

If `tool.uv.sources` is defined in the workspace root, it applies to all members, unless overridden
in the `tool.uv.sources` of a specific member.
```toml title="pyproject.toml"
[tool.uv.workspace]
members = ["packages/*"]
```

A virtual root _must not_ contain a `[project]` table, as the inclusion of a `[project]` table
implies the directory is a package, and thus an explicit root. As such, virtual roots cannot define
their own dependencies; however, they _can_ define development dependencies as in:

```toml title="pyproject.toml"
[tool.uv.workspace]
members = ["packages/*"]

Using `uv init` inside a workspace will add the newly created package to `members`.
[tool.uv]
dev-dependencies = ["ruff==0.5.0"]
```

By default, `uv run` and `uv sync` operates on the workspace root, if it's explicit. For example, in
the above example, `uv run` and `uv run --package albatross` would be equivalent. For virtual
workspaces, `uv run` and `uv sync` instead sync all workspace members, since the root is not a
member itself.

By default, running `uv init` inside an existing package will add the newly created member to the
workspace.

## Common structures
## Workspace sources

There a two main workspace structures: A **root package with helpers** and a **flat workspace**.
Within a workspace, dependencies on workspace members are facilitated via `tool.uv.sources`, as in:

The root workspace layout defines one main package in the root of the repository, with helper
packages in `packages`. In this example `albatross/pyproject.toml` has both a `project` section and
a `tool.uv.workspace` section.
```toml title="pyproject.toml"
[project]
name = "albatross"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["bird-feeder", "tqdm>=4,<5"]

[tool.uv.sources]
bird-feeder = { workspace = true }

[tool.uv.workspace]
members = ["packages/*"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

In this example, the `albatross` package depends on the `bird-feeder` package, which is a member of
the workspace. The `workspace = true` key-value pair in the `tool.uv.sources` table indicates the
`bird-feeder` dependency should be provided by the workspace, rather than fetched from PyPI or
another registry.

Any `tool.uv.sources` definitions in the workspace root apply to all members, unless overridden in
the `tool.uv.sources` of a specific member. For example, given the following `pyproject.toml`:

```toml title="pyproject.toml"
[project]
name = "albatross"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["bird-feeder", "tqdm>=4,<5"]

[tool.uv.sources]
bird-feeder = { workspace = true }
tqdm = { git = "https://github.com/tqdm/tqdm" }

[tool.uv.workspace]
members = ["packages/*"]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

Every workspace member would, by default, install `tqdm` from GitHub, unless a specific member
overrides the `tqdm` entry in its own `tool.uv.sources` table.

## Workspace layouts

In general, there are two common layouts for workspaces, which map to the two kinds of workspace
roots: a **root package with helpers** (for explicit roots) and a **flat workspace** (for virtual
roots).

In the former case, the workspace includes an explicit workspace root, with peripheral packages or
libraries defined in `packages`. For example, here, `albatross` is an explicit workspace root, and
`bird-feeder` and `seeds` are workspace members:

```text
albatross
├── packages
│ ├── provider_a
│ ├── bird-feeder
│ │ ├── pyproject.toml
│ │ └── src
│ │ └── provider_a
│ │ └── bird_feeder
│ │ ├── __init__.py
│ │ └── foo.py
│ └── provider_b
│ └── seeds
│ ├── pyproject.toml
│ └── src
│ └── provider_b
│ └── seeds
│ ├── __init__.py
│ └── bar.py
├── pyproject.toml
Expand All @@ -80,9 +175,8 @@ albatross
└── main.py
```

In the flat layout, all packages are in the `packages` directory, and the root `pyproject.toml`
defines a so-called virtual workspace. In this example `albatross/pyproject.toml` has only a
`tool.uv.workspace` section, but no `project`.
In the latter case, _all_ members are located in the `packages` directory, and the root
`pyproject.toml` comprises a virtual root:

```text
albatross
Expand All @@ -93,32 +187,70 @@ albatross
│ │ └── albatross
│ │ ├── __init__.py
│ │ └── foo.py
│ ├── provider_a
│ ├── bird-feeder
│ │ ├── pyproject.toml
│ │ └── src
│ │ └── provider_a
│ │ └── bird_feeder
│ │ ├── __init__.py
│ │ └── foo.py
│ └── provider_b
│ └── seeds
│ ├── pyproject.toml
│ └── src
│ └── provider_b
│ └── seeds
│ ├── __init__.py
│ └── bar.py
├── pyproject.toml
├── README.md
└── uv.lock
```

In the flat layout, you may still define development dependencies in the workspace root
`pyproject.toml`:
## When (not) to use workspaces

Workspaces are intended to facilitate the development of multiple interconnected packages within a
single repository. As a codebase grows in complexity, it can be helpful to split it into smaller,
composable packages, each with their own dependencies and version constraints.

Workspaces help enforce isolation and separation of concerns. For example, in uv, we have separate
packages for the core library and the command-line interface, enabling us to test the core library
independently of the CLI, and vice versa.

Other common use cases for workspaces include:

- A library with a performance-critical subroutine implemented in an extension module (Rust, C++,
etc.).
- A library with a plugin system, where each plugin is a separate workspace package with a
dependency on the root.

Workspaces are _not_ suited for cases in which members have conflicting requirements, or desire a
separate virtual environment for each member. In this case, path dependencies are often preferable.
For example, rather than grouping `albatross` and its members in a workspace, you can always define
each package as its own independent project, with inter-package dependencies defined as path
dependencies in `tool.uv.sources`:

```toml title="pyproject.toml"
[tool.uv.workspace]
members = ["packages/*"]
[project]
name = "albatross"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["bird-feeder", "tqdm>=4,<5"]

[tool.uv]
dev-dependencies = [
"pytest >=8.3.2,<9"
]
[tool.uv.sources]
bird-feeder = { path = "packages/bird-feeder" }

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
```

This approach conveys many of the same benefits, but allows for more fine-grained control over
dependency resolution and virtual environment management (with the downside that `uv run --package`
is no longer available; instead, commands must be run from the relevant package directory).

Finally, uv's workspaces enforce a single `requires-python` for the entire workspace, taking the
intersection of all members' `requires-python` values. If you need to support testing a given member
on a Python version that isn't supported by the rest of the workspace, you may need to use `uv pip`
to install that member in a separate virtual environment.

!!! note

As Python does not provide dependency isolation, uv can't ensure that a package uses its declared dependencies and nothing else. For workspaces specifically, uv can't ensure that packages don't import dependencies declared by another workspace member.

0 comments on commit f583e8b

Please sign in to comment.