Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow multiple pinned indexes in tool.uv.sources #7769

Merged
merged 1 commit into from
Oct 15, 2024

Conversation

charliermarsh
Copy link
Member

@charliermarsh charliermarsh commented Sep 28, 2024

Summary

This PR lifts the restriction that a package must come from a single index. For example, you can now do:

[project]
name = "project"
version = "0.1.0"
readme = "README.md"
requires-python = ">=3.12"
dependencies = ["jinja2"]

[tool.uv.sources]
jinja2 = [
    { index = "torch-cu118", marker = "sys_platform == 'darwin'"},
    { index = "torch-cu124", marker = "sys_platform != 'darwin'"},
]

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"

[[tool.uv.index]]
name = "torch-cu124"
url = "https://download.pytorch.org/whl/cu124"

The construction is very similar to the way we handle URLs today: you can have multiple URLs for a given package, but they must appear in disjoint forks. So most of the code is just adding that abstraction to the resolver, following our handling of URLs.

Closes #7761.

@charliermarsh charliermarsh added the enhancement New feature or improvement to existing functionality label Sep 28, 2024
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from f2a6d3d to ca30bc3 Compare September 28, 2024 23:29
@charliermarsh charliermarsh marked this pull request as ready for review September 28, 2024 23:29
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch 5 times, most recently from 051f118 to 31a5e8b Compare September 29, 2024 00:14
| ^
Sources can only include a single index source
Resolved 4 packages in [TIME]
error: found duplicate package `jinja2==3.1.3 @ registry+https://download.pytorch.org/whl/cu118`
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This one actually fails:

[project]
name = "project"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = ["jinja2>=3"]

[tool.uv.sources]
jinja2 = [
    { index = "torch-cu118", marker = "sys_platform == 'win32'"},
]

[[tool.uv.index]]
name = "torch-cu118"
url = "https://download.pytorch.org/whl/cu118"

(It succeeds if you use explicit = true on the index.)

The issue is that the sys_platform == 'win32' branch uses the torch-cu118 index explicitly, but then the implied sys_platform != 'win32' branch also resolves to that same index.

@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from 31a5e8b to 495b958 Compare September 29, 2024 01:48
Copy link
Member

@konstin konstin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

YES

Mark this bold in the release notes

crates/uv-resolver/src/resolver/mod.rs Outdated Show resolved Hide resolved
crates/uv-resolver/src/fork_indexes.rs Show resolved Hide resolved
@charliermarsh charliermarsh force-pushed the charlie/index-api branch 3 times, most recently from 1cab0d9 to 1ba4caa Compare September 30, 2024 21:20
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from 495b958 to ae705cb Compare September 30, 2024 21:21
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from ae705cb to 0f92b1f Compare September 30, 2024 22:24
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from 0f92b1f to 3eece5d Compare October 1, 2024 00:48
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from 3eece5d to df669c8 Compare October 1, 2024 00:52
@zanieb
Copy link
Member

zanieb commented Oct 1, 2024

Looks like this needs a docs update though.

@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from df669c8 to 20341ab Compare October 1, 2024 21:24
@charliermarsh charliermarsh force-pushed the charlie/index-api branch 2 times, most recently from 6b44e69 to 23a6de0 Compare October 2, 2024 00:46
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from 20341ab to a4d6408 Compare October 2, 2024 01:12
@chitralverma
Copy link

chitralverma commented Oct 3, 2024

would love to use this. looking forward to a release with this and some updated docs!
any ideas on when we can expect this?

@charliermarsh
Copy link
Member Author

Probably next week just because I'm traveling right now.

@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from a4d6408 to 62db92c Compare October 3, 2024 18:09
@chitralverma
Copy link

Hi @charliermarsh any updates on this one, anything I can help with?

@charliermarsh
Copy link
Member Author

No, it’ll ship on Monday.

@chitralverma
Copy link

chitralverma commented Oct 14, 2024

@charliermarsh can we also publish some new docs for torch installation with this PR?

@charliermarsh charliermarsh force-pushed the charlie/index-api branch 3 times, most recently from 4b0e753 to 9e54908 Compare October 15, 2024 22:07
Base automatically changed from charlie/index-api to main October 15, 2024 22:24
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch 3 times, most recently from cca4018 to a1ff0c1 Compare October 15, 2024 22:39
@charliermarsh charliermarsh force-pushed the charlie/index-api-multiple-indexes branch from a1ff0c1 to c656754 Compare October 15, 2024 22:51
@charliermarsh charliermarsh enabled auto-merge (squash) October 15, 2024 22:51
@charliermarsh charliermarsh merged commit 9a76e47 into main Oct 15, 2024
60 checks passed
@charliermarsh charliermarsh deleted the charlie/index-api-multiple-indexes branch October 15, 2024 22:58
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Oct 18, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.22` -> `0.4.24` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.24`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0424)

[Compare Source](astral-sh/uv@0.4.23...0.4.24)

##### Bug fixes

-   Fix Python executable name in Windows free-threaded Python distributions ([#&#8203;8310](astral-sh/uv#8310))
-   Redact index credentials from lockfile sources ([#&#8203;8307](astral-sh/uv#8307))
-   Respect `UV_INDEX_` rather than `UV_HTTP_BASIC_` as documented ([#&#8203;8306](astral-sh/uv#8306))
-   Improve sources deserialization errors ([#&#8203;8308](astral-sh/uv#8308))

##### Documentation

-   Correct pytorch-to-torch reference in docs ([#&#8203;8291](astral-sh/uv#8291))

### [`v0.4.23`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#0423)

[Compare Source](astral-sh/uv@0.4.22...0.4.23)

This release introduces a revamped system for defining package indexes, as an alternative to the existing pip-style
`--index-url` and `--extra-index-url` configuration options.

You can now define named indexes in your `pyproject.toml` file using the `[[tool.uv.index]]` table:

```toml
[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
```

Packages can be pinned to a specific index via `tool.uv.sources`, to ensure that a given package is installed from the
correct index. For example, to ensure that `torch` is *always* installed from the `pytorch` index:

```toml
[tool.uv.sources]
torch = { index = "pytorch" }

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
```

Indexes can also be marked as `explicit = true` to prevent packages from being installed from that index
unless explicitly pinned. For example, to ensure that `torch` is installed from the `pytorch` index, but all other
packages are installed from the default index:

```toml
[tool.uv.sources]
torch = { index = "pytorch" }

[[tool.uv.index]]
name = "pytorch"
url = "https://download.pytorch.org/whl/cpu"
explicit = true
```

To define an additional index outside a `pyproject.toml` file, use the `--index` command-line argument
(or the `UV_INDEX` environment variable); to replace the default index (PyPI), use the `--default-index` command-line
argument (or `UV_DEFAULT_INDEX`).

These changes are entirely backwards-compatible with the deprecated `--index-url` and `--extra-index-url` options,
which continue to work as before.

See the [Index](https://docs.astral.sh/uv/configuration/indexes/) documentation for more.

##### Enhancements

-   Add index URLs when provided via `uv add --index` or `--default-index` ([#&#8203;7746](astral-sh/uv#7746))
-   Add support for named and explicit indexes ([#&#8203;7481](astral-sh/uv#7481))
-   Add templates for popular build backends ([#&#8203;7857](astral-sh/uv#7857))
-   Allow multiple pinned indexes in `tool.uv.sources` ([#&#8203;7769](astral-sh/uv#7769))
-   Allow users to incorporate Git tags into dynamic cache keys ([#&#8203;8259](astral-sh/uv#8259))
-   Pin named indexes in `uv add` ([#&#8203;7747](astral-sh/uv#7747))
-   Respect named `--index` and `--default-index` values in `tool.uv.sources` ([#&#8203;7910](astral-sh/uv#7910))
-   Update to latest PubGrub version ([#&#8203;8245](astral-sh/uv#8245))
-   Enable environment variable authentication for named indexes ([#&#8203;7741](astral-sh/uv#7741))
-   Avoid showing lower-bound warning outside of explicit lock and sync ([#&#8203;8234](astral-sh/uv#8234))
-   Improve logging during lock errors ([#&#8203;8258](astral-sh/uv#8258))
-   Improve styling of `requires-python` warnings ([#&#8203;8240](astral-sh/uv#8240))
-   Show hint in resolution failure on `Forbidden` (`403`) or `Unauthorized` (`401`) ([#&#8203;8264](astral-sh/uv#8264))
-   Update to latest `cargo-dist` version (includes new installer features) ([#&#8203;8270](astral-sh/uv#8270))
-   Warn when patch version in `requires-python` is implicitly `0` ([#&#8203;7959](astral-sh/uv#7959))
-   Add more context on client errors during range requests ([#&#8203;8285](astral-sh/uv#8285))

##### Bug fixes

-   Avoid writing duplicate index URLs with `--emit-index-url` ([#&#8203;8226](astral-sh/uv#8226))
-   Fix error leading to out-of-bound panic in `uv-pep508` ([#&#8203;8282](astral-sh/uv#8282))
-   Fix managed distributions of free-threaded Python on Windows ([#&#8203;8268](astral-sh/uv#8268))
-   Fix selection of free-threaded interpreters during default Python discovery ([#&#8203;8239](astral-sh/uv#8239))
-   Ignore sources in build requirements for non-source trees ([#&#8203;8235](astral-sh/uv#8235))
-   Invalid cache when adding lower bound to lockfile ([#&#8203;8230](astral-sh/uv#8230))
-   Respect index priority when storing credentials ([#&#8203;8256](astral-sh/uv#8256))
-   Respect relative paths in `uv build` sources ([#&#8203;8237](astral-sh/uv#8237))
-   Narrow what the pip3.<minor> logic drops from entry points. ([#&#8203;8273](astral-sh/uv#8273))

##### Documentation

-   Add some additional notes to `--index-url` docs ([#&#8203;8267](astral-sh/uv#8267))
-   Add upgrade note to README ([#&#8203;7937](astral-sh/uv#7937))
-   Remove note that "only a single source may be defined for each dependency" ([#&#8203;8243](astral-sh/uv#8243))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or improvement to existing functionality
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants