Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Usage of bzlmod causes cache invalidations due to module version being in file path #20997

Closed
v3n opened this issue Jan 23, 2024 · 1 comment
Closed

Usage of bzlmod causes cache invalidations due to module version being in file path #20997

v3n opened this issue Jan 23, 2024 · 1 comment
Assignees
@fmeum
Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P2 We'll consider working on this in future. (Assignee optional) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request

Comments

@v3n
Copy link

v3n commented Jan 23, 2024

Description of the feature request:

Remove module version from the visible module path as it causes frequent cache invalidations when modules updates. This is particularly problematic in C++ where changes in include paths due to module versions can propagate a cache invalidation over most of the build graph; despite no actual dependent files changing.

Which category does this issue belong to?

External Dependency

What underlying problem are you trying to solve with this feature?

No response

Which operating system are you running Bazel on?

Mac OS

What is the output of bazel info release?

release 6.4.0

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

No response

Any other information, logs, or outputs that you want to share?

No response
@meteorcloudy meteorcloudy added the team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. label Jan 23, 2024
@meteorcloudy meteorcloudy changed the title Usage of bzlmod causes cause invalidations due to module version being in file path Usage of bzlmod causes cache invalidations due to module version being in file path Jan 23, 2024
@meteorcloudy meteorcloudy added P2 We'll consider working on this in future. (Assignee optional) area-Bzlmod Bzlmod-specific PRs, issues, and feature requests and removed untriaged labels Jan 23, 2024
This was referenced Jan 25, 2024
Closed
Closed
copybara-service bot pushed a commit that referenced this issue Jan 26, 2024
@fmeum @copybara-github
Fix inconsistent dep graph stubs in Bzlmod tests
27419e3 
These cause failures when relying on `depGraph` containing `ModuleKey`s for all modules, which is the case in production.

Work towards #20997

Closes #21037.

PiperOrigin-RevId: 601822357
Change-Id: Ifcad9d7b73835491c1f1fca975e05834057a6825
bazel-io pushed a commit to bazel-io/bazel that referenced this issue Jan 26, 2024
@fmeum @bazel-io
Fix inconsistent dep graph stubs in Bzlmod tests
caf8679 
These cause failures when relying on `depGraph` containing `ModuleKey`s for all modules, which is the case in production.

Work towards bazelbuild#20997

Closes bazelbuild#21037.

PiperOrigin-RevId: 601822357
Change-Id: Ifcad9d7b73835491c1f1fca975e05834057a6825
@bazel-io bazel-io mentioned this issue Jan 26, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Jan 26, 2024
@bazel-io @fmeum
[7.1.0] Fix inconsistent dep graph stubs in Bzlmod tests (#21085)
855aa90 
These cause failures when relying on `depGraph` containing `ModuleKey`s
for all modules, which is the case in production.

Work towards #20997

Closes #21037.

Commit
27419e3

PiperOrigin-RevId: 601822357
Change-Id: Ifcad9d7b73835491c1f1fca975e05834057a6825

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
copybara-service bot pushed a commit that referenced this issue Feb 5, 2024
@fmeum @copybara-github
Do not store the repository name in RepoSpec
a87a8db 
The name can instead be set centrally in `BzlmodRepoRuleFunction`. This removes some unnecessary glue code and paves the way for rewriting canonical repo names based on knowledge of the full dep graph.

Work towards #20997

Closes #21026.

PiperOrigin-RevId: 604256186
Change-Id: Ia49123f016d1512074b04ec458f017f4bbb88233
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Feb 5, 2024
@fmeum @iancha1992
Do not store the repository name in RepoSpec
4ca917b 
The name can instead be set centrally in `BzlmodRepoRuleFunction`. This removes some unnecessary glue code and paves the way for rewriting canonical repo names based on knowledge of the full dep graph.

Work towards bazelbuild#20997

Closes bazelbuild#21026.

PiperOrigin-RevId: 604256186
Change-Id: Ia49123f016d1512074b04ec458f017f4bbb88233
@iancha1992 iancha1992 mentioned this issue Feb 5, 2024
Closed
iancha1992 pushed a commit to iancha1992/bazel that referenced this issue Feb 5, 2024
@fmeum @iancha1992
Do not store the repository name in RepoSpec
e5bb173 
The name can instead be set centrally in `BzlmodRepoRuleFunction`. This removes some unnecessary glue code and paves the way for rewriting canonical repo names based on knowledge of the full dep graph.

Work towards bazelbuild#20997

Closes bazelbuild#21026.

PiperOrigin-RevId: 604256186
Change-Id: Ia49123f016d1512074b04ec458f017f4bbb88233
@iancha1992 iancha1992 mentioned this issue Feb 5, 2024
Closed
Wyverald pushed a commit that referenced this issue Feb 6, 2024
@fmeum @Wyverald
Do not store the repository name in RepoSpec
9215541 
The name can instead be set centrally in `BzlmodRepoRuleFunction`. This removes some unnecessary glue code and paves the way for rewriting canonical repo names based on knowledge of the full dep graph.

Work towards #20997

Closes #21026.

PiperOrigin-RevId: 604256186
Change-Id: Ia49123f016d1512074b04ec458f017f4bbb88233
@Wyverald Wyverald mentioned this issue Feb 6, 2024
Merged
github-merge-queue bot pushed a commit that referenced this issue Feb 6, 2024
@Wyverald @fmeum
[7.1.0] Do not store the repository name in RepoSpec (#21209)
edd44fd 
The name can instead be set centrally in `BzlmodRepoRuleFunction`. This
removes some unnecessary glue code and paves the way for rewriting
canonical repo names based on knowledge of the full dep graph.

Work towards #20997

Closes #21026.

PiperOrigin-RevId: 604256186
Change-Id: Ia49123f016d1512074b04ec458f017f4bbb88233

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@fmeum fmeum mentioned this issue Feb 6, 2024
Closed
Wyverald pushed a commit that referenced this issue Feb 12, 2024
@fmeum @Wyverald
Omit unique module versions from canonical repo names
3cf373b 
The canonical repository name of a module `rules_foo` is now `rules_foo~` instead of e.g. `rules_foo~1.2.3` if there is only a single version of the module in the entire dep graph. This also applies to overrides, which previously used the canonical repository name `rules_foo~override`.

This improves cacheability of actions across module version changes and also prevents the output base from filling up with outdated module and extension repositories. See the long comment in `ModuleKey#getCanonicalRepoName` for a detailed explanation of why this particular scheme was chosen.

The change includes a bump of the lockfile version as the canonical repo names of label attributes in extension usages are affected.

RELNOTES: The scheme for generating canonical repository names has changed to improve cacheability of actions across dependency version updates. Note that canonical names are not considered to be public API and can change at any time. See https://bazel.build/external/module#repository_names_and_strict_deps for advice on how to avoid hardcoding canonical repository names.

Fixes #20997

Closes #21035.

PiperOrigin-RevId: 605730371
Change-Id: Ica1be1ba5493d3636248a79a6549a0927021bef9
@Wyverald Wyverald mentioned this issue Feb 12, 2024
Merged
Wyverald pushed a commit that referenced this issue Feb 12, 2024
@fmeum @Wyverald
Omit unique module versions from canonical repo names
5e03794 
The canonical repository name of a module `rules_foo` is now `rules_foo~` instead of e.g. `rules_foo~1.2.3` if there is only a single version of the module in the entire dep graph. This also applies to overrides, which previously used the canonical repository name `rules_foo~override`.

This improves cacheability of actions across module version changes and also prevents the output base from filling up with outdated module and extension repositories. See the long comment in `ModuleKey#getCanonicalRepoName` for a detailed explanation of why this particular scheme was chosen.

The change includes a bump of the lockfile version as the canonical repo names of label attributes in extension usages are affected.

RELNOTES: The scheme for generating canonical repository names has changed to improve cacheability of actions across dependency version updates. Note that canonical names are not considered to be public API and can change at any time. See https://bazel.build/external/module#repository_names_and_strict_deps for advice on how to avoid hardcoding canonical repository names.

Fixes #20997

Closes #21035.

PiperOrigin-RevId: 605730371
Change-Id: Ica1be1ba5493d3636248a79a6549a0927021bef9
github-merge-queue bot pushed a commit that referenced this issue Feb 13, 2024
@Wyverald @fmeum
[7.1.0] Omit unique module versions from canonical repo names (#21316)
1991f52 
The canonical repository name of a module `rules_foo` is now
`rules_foo~` instead of e.g. `rules_foo~1.2.3` if there is only a single
version of the module in the entire dep graph. This also applies to
overrides, which previously used the canonical repository name
`rules_foo~override`.

This improves cacheability of actions across module version changes and
also prevents the output base from filling up with outdated module and
extension repositories. See the long comment in
`ModuleKey#getCanonicalRepoName` for a detailed explanation of why this
particular scheme was chosen.

The change includes a bump of the lockfile version as the canonical repo
names of label attributes in extension usages are affected.

RELNOTES: The scheme for generating canonical repository names has
changed to improve cacheability of actions across dependency version
updates. Note that canonical names are not considered to be public API
and can change at any time. See
https://bazel.build/external/module#repository_names_and_strict_deps for
advice on how to avoid hardcoding canonical repository names.

Fixes #20997

Closes #21035.

PiperOrigin-RevId: 605730371
Change-Id: Ica1be1ba5493d3636248a79a6549a0927021bef9

Co-authored-by: Fabian Meumertzheim <fabian@meumertzhe.im>
@iancha1992
Copy link
Member

A fix for this issue has been included in Bazel 7.1.0 RC1. Please test out the release candidate and report any issues as soon as possible. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fmeum fmeum

Labels
area-Bzlmod Bzlmod-specific PRs, issues, and feature requests P2 We'll consider working on this in future. (Assignee optional) team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. type: feature request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Omit unique module versions from canonical repo names fmeum/bazel
6 participants
@v3n @meteorcloudy @fmeum @sgowroji @iancha1992 @rakshitasingh05