Only garbage collect action cache entries

[buchgr]

The cache should only garbage collect action cache entries based on last access time. CAS entries should have a reference count and only be collected if they are no longer referenced by any cache entry.

[bayareabear]

Can we make an assumption that one CAS entry is only referenced by one ac entry, so that when we delete ac entry, we just delete all the CAS entries referenced by that ac entry?

[buchgr]

Can we make an assumption that one CAS entry is only referenced by one ac entry, so that when we delete ac entry, we just delete all the CAS entries referenced by that ac entry?

No that's generally not true. A blob is often referenced by thousands of action cache entries. I am currently implementing this feature and adding a reference count to each CAS entry. So we'll only evict a CAS entry when its no longer referenced.

[bayareabear]

Then when PUT AC/CAS to server, Is the following sequence always guaranteed to ensure the server correctness:

1. All the CAS entries referred by a AC are PUT to server first
2. Then the AC entry is PUT to server

[bayareabear]

Also, is it possible to let Bazel back off when it GET an AC entry returns 200 but the corresponding CAS is not found in the server? I found when this happens, bazel build may fail. The Bazel build requirement that CAS entry referred by AC entry always exist in the server causes two drawbacks:

1. We have to keep track the reference count of CAS object for garbage collection
2. For horizontal scale and load balancing, consistency among servers can be hard to maintain
   The ideal Bazel behavious is following:
   Every CAS entry can be tracked back to the corresponding AC that generate it. In case a CAS entry referred by an AC is missing, back track the AC entry and re-run the AC to regenerate the missed CAS entry. This immediately fix the two drawbacks mentioned above.

[mostynb]

Also, is it possible to let Bazel back off when it GET an AC entry returns 200 but the corresponding CAS is not found in the server?

This is implemented in my gRPC WiP in #97, and I filed #106 for a backwards-compatible solution for http.

@bayareabear ^