Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for dynamic cache keys #7136

Merged
merged 9 commits into from
Sep 9, 2024
Merged

Conversation

charliermarsh
Copy link
Member

@charliermarsh charliermarsh commented Sep 6, 2024

Summary

This PR adds a more flexible cache invalidation abstraction for uv, and uses that new abstraction to improve support for dynamic metadata.

Specifically, instead of relying solely on a timestamp, we now pass around a CacheInfo struct which (as of now) contains Option<Timestamp> and Option<Commit>. The CacheInfo is saved in dist-info as uv_cache.json, so we can test already-installed distributions for cache validity (along with testing cached distributions for cache validity).

Beyond the defaults (pyproject.toml, setup.py, and setup.cfg changes), users can also specify additional cache keys, and it's easy for us to extend support in the future. Right now, cache keys can either be instructions to include the current commit (for setuptools_scm and similar) or file paths (for hatch-requirements-txt and similar):

[tool.uv]
cache-keys = [{ file = "requirements.txt" }, { git = true }]

This change should be fully backwards compatible.

Closes #6964.

Closes #6255.

Closes #6860.

@charliermarsh charliermarsh marked this pull request as ready for review September 6, 2024 21:11
@charliermarsh charliermarsh added enhancement New feature or improvement to existing functionality cache Caching of packages and metadata labels Sep 6, 2024
@charliermarsh
Copy link
Member Author

Going to move the #7028 part into its own commit, since it's largely independent.

@charliermarsh charliermarsh changed the title Add support for dynamic cache keys and build settings Add support for dynamic cache keys Sep 6, 2024
@charliermarsh charliermarsh force-pushed the charlie/dynamic-cache branch 2 times, most recently from 7213864 to ebc1e30 Compare September 6, 2024 21:44
=== "uv.toml"

```toml

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change


```toml
[tool.uv]
cache-keys = [{ file = "requirements.txt" }, { git = true }]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that we always read the default three files, should this be called extend-cache-keys, mirroring ruff's extend- settings?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't feel strongly about this, but we don't use extend- at all in uv yet and you won't be able to change the base cache keys so I don't think it's necessary here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not particular on the specific naming, but i think it's an odd design that we expose a cache-keys setting that then doesn't let you configure the cache keys, but add files/git to our immutable defaults.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we make our current settings a default, and this overrides them?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd find that more intuitive. It's a bit annoying that you have to re-specify pyproject.toml and/or setup.py, which i assumed that's what you wanted to avoid with that design; but personally i like just specifying all relevant files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Certainly an option, but are there cases where you would actually want to ignore changes to those other files?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess one upside is that we could document the default more clearly...? I'm torn here though, I guess I don't feel strongly either way.

crates/uv-cache-info/src/cache_info.rs Show resolved Hide resolved
crates/uv-cache-info/src/cache_info.rs Show resolved Hide resolved
crates/uv-cache-info/src/cache_info.rs Show resolved Hide resolved
crates/uv-cache-info/src/cache_info.rs Show resolved Hide resolved
crates/uv-cache-info/src/cache_info.rs Show resolved Hide resolved
crates/uv/tests/pip_install.rs Show resolved Hide resolved
crates/install-wheel-rs/Readme.md Outdated Show resolved Hide resolved
crates/uv-cache-info/src/commit_info.rs Outdated Show resolved Hide resolved
@charliermarsh charliermarsh force-pushed the charlie/dynamic-cache branch 2 times, most recently from 22ebf0b to 54fa5cd Compare September 9, 2024 18:28
Copy link

codspeed-hq bot commented Sep 9, 2024

CodSpeed Performance Report

Merging #7136 will not alter performance

Comparing charlie/dynamic-cache (d366c7e) with main (9a7262c)

Summary

✅ 14 untouched benchmarks

@sbidoul
Copy link

sbidoul commented Sep 9, 2024

Ooh, this is interesting. ❤️
Could file accept a glob pattern?

@charliermarsh
Copy link
Member Author

I don't see why not, though it will get slower as you add more files hah.

@charliermarsh
Copy link
Member Author

Can you give an example of how you'd use it? Just curious.

@sbidoul
Copy link

sbidoul commented Sep 9, 2024

In Odoo projects (using the hatch-odoo build backend), we have dynamic dependencies computed from Odoo addons manifests. So I'd use odoo/addons/*/__manifest__.py.

@charliermarsh charliermarsh enabled auto-merge (squash) September 9, 2024 20:18
@charliermarsh charliermarsh merged commit 4f23491 into main Sep 9, 2024
57 checks passed
@charliermarsh charliermarsh deleted the charlie/dynamic-cache branch September 9, 2024 20:19
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Sep 11, 2024
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [astral-sh/uv](https://github.com/astral-sh/uv) | patch | `0.4.4` -> `0.4.9` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>astral-sh/uv (astral-sh/uv)</summary>

### [`v0.4.9`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#049)

[Compare Source](astral-sh/uv@0.4.8...0.4.9)

##### Enhancements

-   Add support for managed Python 3.13 ([#&#8203;7263](astral-sh/uv#7263))
-   Upgrade managed CPython versions to latest patch releases ([#&#8203;7263](astral-sh/uv#7263))
-   Allow setting a target version for `uv self update` ([#&#8203;7252](astral-sh/uv#7252))
-   Create `py.typed` files during `uv init --lib` ([#&#8203;7232](astral-sh/uv#7232))
-   Add a dedicated error for packages that fail due to `distutils` deprecation ([#&#8203;7239](astral-sh/uv#7239))
-   Improve error message when requested Python version is unsupported ([#&#8203;7269](astral-sh/uv#7269))
-   Add `uv run --no-sync` ([#&#8203;7192](\(https://github.com/astral-sh/uv/pull/7192\))

##### Bug fixes

-   Avoid updating `pyproject.toml` offsets on non-add edits ([#&#8203;7262](astral-sh/uv#7262))
-   Invalidate cache when `--config-settings` change ([#&#8203;7139](astral-sh/uv#7139))
-   Remove workspace root for single-member workspace with `uv export` ([#&#8203;7254](astral-sh/uv#7254))

### [`v0.4.8`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#048)

[Compare Source](astral-sh/uv@0.4.7...0.4.8)

##### Enhancements

-   Add support for dynamic cache keys ([#&#8203;7136](astral-sh/uv#7136))
-   Allow `.dist-info` names with dashes for post releases ([#&#8203;7208](astral-sh/uv#7208))
-   Use type hints in code from `uv init` ([#&#8203;7225](astral-sh/uv#7225))
-   Treat `.tgz` the same as `.tar.gz` ([#&#8203;7201](astral-sh/uv#7201))
-   Direct users towards `uv venv` to create a virtual environment ([#&#8203;7188](astral-sh/uv#7188))
-   Improve error message for uv init already init-ed ([#&#8203;7198](astral-sh/uv#7198))

##### Performance

-   Avoid batch prefetching for un-optimized registries ([#&#8203;7226](astral-sh/uv#7226))
-   Avoid iteration for singleton selections ([#&#8203;7195](astral-sh/uv#7195))

##### Bug fixes

-   Avoid extra newlines in debug logging for source builds ([#&#8203;7174](astral-sh/uv#7174))
-   Prune unreachable packages from `--universal` output ([#&#8203;7209](astral-sh/uv#7209))
-   Respect exclusion when collecting workspace members ([#&#8203;7175](astral-sh/uv#7175))
-   Use path file instead of `sitecustomize.py` ([#&#8203;7161](astral-sh/uv#7161))
-   Replace incorrect `--source` and `--binary` flags with correct `--sdist` and `--wheel` flags in `uv build` ([#&#8203;7156](astral-sh/uv#7156))

##### Documentation

-   Document support for `UV_INSTALL_DIR` ([#&#8203;7107](astral-sh/uv#7107))
-   List all supported sdist formats ([#&#8203;7168](astral-sh/uv#7168))

### [`v0.4.7`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#047)

[Compare Source](astral-sh/uv@0.4.6...0.4.7)

##### Enhancements

-   Add `--no-emit-project` and friends to `uv export` ([#&#8203;7110](astral-sh/uv#7110))
-   Add `--output-file` to `uv export` ([#&#8203;7109](astral-sh/uv#7109))
-   Prune unused source distributions from the cache in `uv cache prune` ([#&#8203;7112](astral-sh/uv#7112))
-   Take intersection of constraint and requirements hashes ([#&#8203;7108](astral-sh/uv#7108))

##### Performance

-   Skip metadata fetch for `--no-deps` and `pip sync` ([#&#8203;7127](astral-sh/uv#7127))

##### Bug fixes

-   Avoid panicking when encountering an invalid Python version during `uv python list` ([#&#8203;7131](astral-sh/uv#7131))
-   Write trailing newline to `.python-version` files ([#&#8203;7140](astral-sh/uv#7140))

### [`v0.4.6`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#046)

[Compare Source](astral-sh/uv@0.4.5...0.4.6)

##### Enhancements

-   Accept `--build-constraints` in `uv build` ([#&#8203;7085](astral-sh/uv#7085))
-   Add `--require-hashes` and `--verify-hashes` to `uv build` ([#&#8203;7094](astral-sh/uv#7094))
-   Add `--show-version-specifiers` to `uv tool list` ([#&#8203;7050](astral-sh/uv#7050))
-   Respect hashes in constraints files ([#&#8203;7093](astral-sh/uv#7093))
-   Upgrade installer scripts ([#&#8203;7092](astral-sh/uv#7092))
-   Allow specifying multiple packages in `uv tool upgrade` and `uninstall` ([#&#8203;7037](astral-sh/uv#7037))
-   Sort by implementation in `uv python list` ([#&#8203;6918](astral-sh/uv#6918))

##### Bug fixes

-   Invalidate lockfile when member versions change ([#&#8203;7102](astral-sh/uv#7102))
-   Strip fragments from direct source URLs in lockfile ([#&#8203;7061](astral-sh/uv#7061))
-   Support `--no-build` and `--no-binary` in `uv sync` et al ([#&#8203;7100](astral-sh/uv#7100))
-   Use distribution hash over registry hash ([#&#8203;7060](astral-sh/uv#7060))
-   Fix inverted log message ([#&#8203;7063](astral-sh/uv#7063))
-   Adjust Docker `ENTRYPOINT` and `CMD` for inherited images ([#&#8203;7054](astral-sh/uv#7054))

##### Documentation

-   Add winget to installers ([#&#8203;7088](astral-sh/uv#7088))
-   Document how to disable path modifications during install ([#&#8203;7090](astral-sh/uv#7090))
-   Document how to manually update locked package version ([#&#8203;7083](astral-sh/uv#7083))
-   Document official `setup-uv` action ([#&#8203;7056](astral-sh/uv#7056))
-   Update docs on `.python-version` file ([#&#8203;7051](astral-sh/uv#7051))

### [`v0.4.5`](https://github.com/astral-sh/uv/blob/HEAD/CHANGELOG.md#045)

[Compare Source](astral-sh/uv@0.4.4...0.4.5)

##### Enhancements

-   Implement `uv build` ([#&#8203;6895](astral-sh/uv#6895))
-   Add `--package` support to `uv build` ([#&#8203;6990](astral-sh/uv#6990))
-   Prune unreachable packages from lockfile ([#&#8203;6959](astral-sh/uv#6959))
-   Prune unreachable wheels from lockfile ([#&#8203;6961](astral-sh/uv#6961))
-   Show build output by default in `uv build` ([#&#8203;6912](astral-sh/uv#6912))
-   Support `uv build --wheel` from source distributions ([#&#8203;6898](astral-sh/uv#6898))
-   Use the root project name for the project virtual environment prompt ([#&#8203;7021](astral-sh/uv#7021))

##### Bug fixes

-   Fix handling of inline optional dependencies in `uv add` ([#&#8203;7023](astral-sh/uv#7023))
-   Reflect exit code in `uv tool run` and `uv run` ([#&#8203;6994](astral-sh/uv#6994))
-   Revert `pyproject.toml` modifications on Ctrl-C ([#&#8203;7024](astral-sh/uv#7024))
-   Rollback `pyproject.toml` changes on all errors ([#&#8203;7022](astral-sh/uv#7022))
-   Use correct ordering semantics for narrowing upper-bounded Python requirements ([#&#8203;7031](astral-sh/uv#7031))
-   Fix segfault in Windows trampolines ([#&#8203;6955](astral-sh/uv#6955))
-   Remove unused `__future__.annotations` import in `_virtualenv.py` ([#&#8203;6996](astral-sh/uv#6996))

##### Documentation

-   Add documentation for `uv build` ([#&#8203;6991](astral-sh/uv#6991))
-   Add note to `extra` and `all-extras` in `uv sync` help ([#&#8203;7013](astral-sh/uv#7013))
-   Add project docs for `project.scripts` ([#&#8203;7010](astral-sh/uv#7010))
-   Fix available Docker image tag rendering and shorten list ([#&#8203;7017](astral-sh/uv#7017))
-   Touchup to the project environment config section ([#&#8203;7038](astral-sh/uv#7038))
-   Clarify precedence of `uv.toml` ([#&#8203;6986](astral-sh/uv#6986))
-   Fix available Docker tags for `-slim` variants ([#&#8203;7041](astral-sh/uv#7041))

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy40NDAuNyIsInVwZGF0ZWRJblZlciI6IjM3LjQ0MC43IiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cache Caching of packages and metadata enhancement New feature or improvement to existing functionality
Projects
None yet
4 participants