Smarter caching for big workspaces by default

[thomaseizinger]

On large Rust repositories, it is easy to exceed GitHub's default cache limit of 10GB. I've recently been working on optimising our CI pipeline over at rust-libp2p. Here is what I landed on:

Our CI fans out testing into one job per crate. Applying this action naively completely busts the cache limit of Github because we would be saving and restoring 20+ caches per workflow run.

To combat this, I created a "cache factory" workflow that only runs on master and compiles the entire workspace: https://github.com/libp2p/rust-libp2p/blob/master/.github/workflows/cache-factory.yml. It is a bit more complicated in our case because we have different caches for the MSRV check and the actual testing but you get the overall idea.

More importantly, we still have other workflows (clippy, rustdoc, cross-compile to wasm etc) that always run on the entire workspace. Those use this action as is. That still creates a lot of caches, hence I am now adding

with:
  save-if: ${{ github.ref == 'refs/head/master' }}

to all cache invocations: libp2p/rust-libp2p#3249

Do you have any thoughts on that? I think that could actually be a safe default to add this action.

[Swatinem]

I believe it something to recommend and call out in the README.

However there certainly are usecases where you wouldn’t want this, or rather where you would have to put more logic into it. Things like long living release or feature branches.

[thomaseizinger]

Okay thanks! I'll leave this open then until the docs are updated! If you can give me a rough structure how you'd like it to be added to the README, I can have a go at a patch.

[Swatinem]

I don’t really have a preference. From your perspective as a user, where would you have expected this?

[thomaseizinger]

I don’t really have a preference. From your perspective as a user, where would you have expected this?

Perhaps a "Big workspaces" or a "Help! My caches are overflowing" section in the README is sufficient?

[thomaseizinger]

Extending the docs for save-if might also be a good idea!

[Swatinem]

Updated readme/docs in 3312b3a