Our CI suffers from cache thrashing #9850
Comments
|
The problem is that cabal does a fair bit of caching (in the form or the store) and GHA caches cannot be updated. The "trick" I mention in the comment you link is just what GitHub recommends to do to update a cache. See https://github.com/actions/cache/blob/main/tips-and-workarounds.md#update-a-cache. The line right after the one you linked is If a cache with the current gha does not exist (as is expected and intentional), GHA will pick the most recent one matching the restore key and create a new one with the current sha. The older cache entries will fall in disuse and will eventually be evicted. If a cache key is present, GHA will restore that but it won't update it or save it again. Of the top of my head, an improvement would be to use separate caches for the store and dist-newstyle. They are different thing that change at different speed. What are you ideas? PS: I don't think we can protect against GitHub (or someone) deleting all the caches. We just do too many builds. We should cull them. |
|
Yes, I understand the mechanism with the I suggest the following strategy: Only write new big caches when CI runs on In practice, we separate |
It sounds like a good plan. I don't think we had separate I still think store and dist-newstyle deserve separate caches. While the store can follow what you propose; dist-newstyle could be cached per branch? Would that make sense? |
At this point in time, GH has evicted all our CI caches of our default branch except one:
This effectively means our caching isn't working in a good way. Because if
masteris bare of caches, every PR has to start building all the dependencies from scratch.Caching has to be engineered so that the latest
mastercaches remain given the expected amount of PRs that are active in parallel (and generate their own caches).In PRs #9845 and #9849 I have tried to do sensible cache engineering so that a new cache is only written if dependencies actually got updated.
However, our main workhorse
validate.ymlsimply creates a new cache on every run (for each OS and GHC it runs) by including thegithub.shain the cache key.cabal/.github/workflows/validate.yml
Line 111 in 7e085fa
With each cache around 300MB, you can do simple arithmetics now to see that our 10GB limit is busted all the time (even 30 GB wouldn't make much of a difference).
E.g. these are the 46 caches by one currently active PR:
Including
github.shain the cache key is a nice solution to cache aging for small projects that have effectively infinite cache space (e.g. if your caches are 10MB and you have 10GB available), but for sizeable project likecabalthis isn't a solution.For
AgdaI spent days in cache engineering so that the goals are met (staying in the limit even with a couple of parallel PRs).The current caching philosophy seems to originate from https://github.com/haskell/cabal/pull/7952/files#r807953232
The text was updated successfully, but these errors were encountered: