-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Garbage collection for cache directory #71
Comments
Hi @aomarks - this issue piqued my curiosity. I am wondering what the current status is? I'm interested in giving it a shot, but it has been sitting for awhile so I figured I would check before digging in. |
Hi @joshkraft! Contributions would be more than welcome. Lately I've been wondering if local caching is something we really want in general. One thing it's nice for is when switching back and forth between git branches, or undos. But I'm not sure the complexity and additional latency of maintaining the local cache is worthwhile. What do you think? Have you been relying on local caching? Did you find that the cache folder was getting too big? |
That is a great question. I actually just started using wireit, so unfortunately I have no real opinion on the utility of the local caching. If the goal is keeping additional complexity/latency low, one interesting approach I noticed in another project is to inject some randomness into the process. Rather than keeping track of every get/put to the cache, each execution calls a Cache cleaning could be done in a number of ways:
I am curious if you think this approach would be a good fit here? I'm happy to do some experimentation and see what it would look like in this project. |
Hi @aomarks, I've had a chance to start using Wireit personally and I have found local caching to be useful, as someone who is a serial branch switcher. I did some POC work implementing a few different eviction mechanisms. My guess is that the additional latency introduced here would largely be driven by the time it takes to delete evicted entries from the cache. I'm curious if you have suggestions on some parameters I can use to benchmark the different approaches? For example:
Also, feel free to let me know if adding this functionality doesn't align with the project's goals. I'm happy to investigate some options and report back with benchmarking results, but if the value added here isn't worth the added complexity I'm happy to direct my effort towards another issue. |
How is the mangement of the default cache and the cache that GitHub Actions might you different currently? In particular, I'm leveraging CircleCI, for which I actually turn off the default CI cache mechanism, because I couldn't get it to actually build/save a cache in that context, however that cache does grow indefintiely. Is there something about the GH Actions cache approach that could be leveraged in "local" mode? If not, then raising the priority of this issue would be really nice to see and support expanded consumption of Wireit across projects I work with at Adobe. |
I do think this is helpful. for cache size, one idea would be for the size target to be a multiple of the size of the live entries in the cache. something like 3x the size, to comfortably fit a few branches (likely more than 3, because they'll share many cache entries) WDYT about a design for this @aomarks? I'm thinking a |
Yeah, I do think that we should either implement garbage collection, or get rid of local caching. I am a little unsure if local caching is worthwhile, so I'd be interested to hear @joshkraft and @rictic 's thoughts on that. Remote caching for CI is clearly useful, and it does its own GC. Local caching feels a bit less useful, and it comes with a cost (latency & storage).
That sounds cool. Are you thinking that it's up the user to run the vacuum command? Would we also run it automatically? Or maybe prompt the user to run it sometimes? |
@aomarks in a large monorepo like |
Yeah, start manual, and then as we gather experience and performance info we could run it automatically sometimes (maybe randomly, every Nth run?) |
+1, I also enjoy having the local caching when working in beefy monorepos. Starting with the manual approach seems like a good idea before weaving GC deeper into the workflow. On the UX side, how would users invoke this command manually? Are we thinking something like |
Maybe it could be similar to
|
In the short term, I was personally thinking that a simple solution would be to just have the vacuum/eviction command work at the project level, cleaning up all caches present in the As far as ongoing cache management, I'm intrigued by the idea of basing size on a multiple of live entries. I'm also wondering if having some sort of user-defined TTL (w/ a sane default) would effective and perhaps simpler... in my use of wireit, I've found that older cache entries are rarely used, though 'older' is relative so it might be nice to have a configurable value. |
We've currently got developers with multi-gigabyte caches so I'm 100% on board with a pruning feature.
The configurability of
I like On the subject of how to handle eviction, it's probably fine to start with something time-based to at least get a solution in-place. We could go back and forth on how best to decide what to evict but ultimately this can be changed since the internals aren't as important to consumers as the fact that it evicts the cache at all. To add to the chaos of how to evict though, what if we took the (configurable) branch into consideration? It's very likely the case that you'd want the cache of your |
Is there likely to be any movement on this soon? I notice the issue is 2 years old (nearly) and am getting increasingly frustrated with remembering to delete my wireit cache every few weeks to avoid it gobbling my whole hard drive. No worries if it's not a priority. Just trying to get a feel for fix timelines. |
The
.wireit/<script>/cache
directory currently can grow indefinitely. We should implement a garbage collection strategy to cap the size of this directory.An LRU cache with a configurable maximum number of entries seems like what we want.
We will want to make sure we have an efficient way to maintain the cache hit rate data which scales well with the size of the cache. We will probably want some kind of on-disk index file that lets us read/write cache hit rates efficiently, to determine which cache entry needs to be deleted when the cap is hit. A doubly-linked-list implemented in the filesystem itself with symlinks (or just files containing SHAs) could also be an interesting way to do this.
The text was updated successfully, but these errors were encountered: