-
-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: cache npm metadata #5491
Merged
Merged
feat: cache npm metadata #5491
Changes from 27 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
d09c0d2
feat: cache npm metadata
paul-soporan 9cd8119
Merge branch 'master' into paul/feat/cache-npm-metadata
paul-soporan f0aaee3
Merge branch 'master' into paul/feat/cache-npm-metadata
paul-soporan c577e8b
refactor: tweaks
paul-soporan 175f1a1
refactor: more tweaks
paul-soporan dba9140
refactor: even more tweaks
paul-soporan 7ec0583
chore: versions
paul-soporan 2deff2a
chore: changelog
paul-soporan caa8e9c
test: update tests
paul-soporan a5ff389
chore: update benchmarks
paul-soporan a1bf470
chore: add todos
paul-soporan 5df66c1
perf: trim stored metadata
paul-soporan f3fc013
refactor: don't export the cache version
paul-soporan 9e42027
refactor: drop unnecessary `as Filename`s
paul-soporan f6fbaa5
fix: add name to stored metadata
paul-soporan 5a0e7c5
feat: automatically derive cache key
paul-soporan 7f7bde8
refactor: slice the cache key
paul-soporan 65b4138
test: update tests
paul-soporan 30a97dd
chore: fix benchmark script
paul-soporan 7cc1f1b
Merge branch 'master' into paul/feat/cache-npm-metadata
paul-soporan a82c891
refactor: remove redundant check
paul-soporan 47ab924
feat: disable metadata cache during hardened mode
paul-soporan 3efc7a9
fix: operator precedence
paul-soporan ede7a2a
refactor: final tweaks
paul-soporan 5268b7a
refactor: use return
paul-soporan c630d25
fix: ensure metadata cache is atomic
paul-soporan d838e3b
refactor: rename for consistency
paul-soporan dd07bf4
Apply suggestions from code review
paul-soporan e662c82
Revert "Apply suggestions from code review"
paul-soporan 6778080
Merge branch 'master' into paul/feat/cache-npm-metadata
paul-soporan 2b1e1ec
refactor: remove `deprecated` field
paul-soporan 07b4ceb
refactor: move `npm-metadata` folder to `metadata/npm`
paul-soporan 7e09021
refactor: use single argument
paul-soporan 600c358
Merge branch 'master' into paul/feat/cache-npm-metadata
arcanis File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
Large diffs are not rendered by default.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,39 @@ | ||
releases: | ||
"@yarnpkg/cli": minor | ||
"@yarnpkg/core": minor | ||
"@yarnpkg/fslib": minor | ||
"@yarnpkg/plugin-npm": minor | ||
|
||
declined: | ||
- "@yarnpkg/plugin-compat" | ||
- "@yarnpkg/plugin-constraints" | ||
- "@yarnpkg/plugin-dlx" | ||
- "@yarnpkg/plugin-essentials" | ||
- "@yarnpkg/plugin-exec" | ||
- "@yarnpkg/plugin-file" | ||
- "@yarnpkg/plugin-git" | ||
- "@yarnpkg/plugin-github" | ||
- "@yarnpkg/plugin-http" | ||
- "@yarnpkg/plugin-init" | ||
- "@yarnpkg/plugin-interactive-tools" | ||
- "@yarnpkg/plugin-link" | ||
- "@yarnpkg/plugin-nm" | ||
- "@yarnpkg/plugin-npm-cli" | ||
- "@yarnpkg/plugin-pack" | ||
- "@yarnpkg/plugin-patch" | ||
- "@yarnpkg/plugin-pnp" | ||
- "@yarnpkg/plugin-pnpm" | ||
- "@yarnpkg/plugin-stage" | ||
- "@yarnpkg/plugin-typescript" | ||
- "@yarnpkg/plugin-version" | ||
- "@yarnpkg/plugin-workspace-tools" | ||
- vscode-zipfs | ||
- "@yarnpkg/builder" | ||
- "@yarnpkg/doctor" | ||
- "@yarnpkg/extensions" | ||
- "@yarnpkg/libzip" | ||
- "@yarnpkg/nm" | ||
- "@yarnpkg/pnp" | ||
- "@yarnpkg/pnpify" | ||
- "@yarnpkg/sdks" | ||
- "@yarnpkg/shell" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm thinking perhaps
<global>/cache/metadata
might be a better location, what do you think? Actions that already cache the<global>/cache
folder would benefit from that out of the box, rather than having to add a new path (or cache the whole<global>
, which may not be as common).I'd also suggest moving the
npm-
prefix from the folder to each individual file name, by consistency with the cache itself (which contains files from the npm, git, http fetchers, etc).There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ping on this comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(Sorry for the delay.)
I don't really like the idea, the mirror and the npm metadata cache are 2 separate things, and merging them in a single folder would lead to confusion. We've also got
index
which would have to be moved tocache
too with this line of thinking.This would just complicate things like
yarn cache clean
, where--mirror
would have to stop removing the entire folder and just remove the archives.I'd prefer all of them to be in separate folders. One thing we could do would be to move the mirror to
<global>/cache/mirror
in addition to what you proposed, then I'd be fine with it and it would still benefit from existing caching (but it would look a bit weird to have the global cache in<global>/cache/mirror
, but I guess that's just the consequence of the mirror and the global cache being the same thing).I'd rather not. It would just give the illusion of consistency.
The cache can do it because it's the sole source that controls that folder.
The npm metadata cache is supposed to be specific to the
plugin-npm
resolvers - they are the only ones that populate and control it.Moving the prefix to the filenames and having the folder called just
metadata
gives the illusion that the folder is a generalized metadata cache. And it could be indeed, if other resolvers ever need to cache metadata in it, but it would have different shapes (and possibly different filename formats) based on the resolvers that cache the data and that might lead to confusion.Edit: The files also wouldn't be in a single folder like the cache, since e.g. for npm we have
npm-metadata/<hash>/<registry>/lodash.json
.It would also have to be controlled by the core, which would be tasked with automatically generating the paths to ensure that no 2 resolvers accidentally use the same path (and also to make it possible for us to change the metadata cache path in the future).
In addition, we're not even certain that the current kind of cache is better than a monolithic one, that's something I'm open to experimenting with in the future.
That's why I think that opening the folder to anything but the npm resolvers is not something I want to do yet, if ever.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔 Thought more about it and I'd be open to making it
<global>/metadata/npm/<hash>/<registry>/lodash.json
.This way, we still have a common metadata folder but we make it clear that each resolver has to manage it manually.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That seems reasonable. I like the idea of moving the cache into cache/mirror (that said, I think we could also just rename --mirror into -g for the same effect; I think I'd like this even better 🤔).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🤔 I still think I'd prefer to keep the mirror, metadata, and index separate.
We could have a
-g
too but I prefer to retain the granularity.yarn cache clean
will become interactive in a future PR anyways (just like we discussed some time ago), and it will clean everything by default.