-
-
Notifications
You must be signed in to change notification settings - Fork 98
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Smarter caching for big workspaces by default #95
Comments
I believe it something to recommend and call out in the README. However there certainly are usecases where you wouldn’t want this, or rather where you would have to put more logic into it. Things like long living release or feature branches. |
Okay thanks! I'll leave this open then until the docs are updated! If you can give me a rough structure how you'd like it to be added to the README, I can have a go at a patch. |
I don’t really have a preference. From your perspective as a user, where would you have expected this? |
Perhaps a "Big workspaces" or a "Help! My caches are overflowing" section in the README is sufficient? |
Extending the docs for |
Updated readme/docs in 3312b3a |
On large Rust repositories, it is easy to exceed GitHub's default cache limit of 10GB. I've recently been working on optimising our CI pipeline over at
rust-libp2p
. Here is what I landed on:Our CI fans out testing into one job per crate. Applying this action naively completely busts the cache limit of Github because we would be saving and restoring 20+ caches per workflow run.
To combat this, I created a "cache factory" workflow that only runs on master and compiles the entire workspace: https://github.com/libp2p/rust-libp2p/blob/master/.github/workflows/cache-factory.yml. It is a bit more complicated in our case because we have different caches for the MSRV check and the actual testing but you get the overall idea.
More importantly, we still have other workflows (clippy, rustdoc, cross-compile to wasm etc) that always run on the entire workspace. Those use this action as is. That still creates a lot of caches, hence I am now adding
to all cache invocations: libp2p/rust-libp2p#3249
Do you have any thoughts on that? I think that could actually be a safe default to add this action.
The text was updated successfully, but these errors were encountered: