Multiple Priority Ordered Caches

[adamgordonbell]

Currently, when using explicit cache a single cache image can be specified to read from. It could be useful to have a priority-ordered list of images to check for cache hits. @mattste provided this use case:

Use Case

I'm attempting to do pull request previews where each pull request has its own Docker image. For pull requests that update a dependency (such as a package.json or mix.lock), these builds can be super slow.

Current Options

Per-branch cache
The suggested fix would be to have the pull request's branch use its own cache instead of using the main branch cache. earthly --remote-cache=us-west1-docker.pkg.dev/my-org/my-repo/build-cache:cache-$branch-name-v1 --push +build.

The downside of this approach is that it can be very slow for the very first commit's build as it has to start from nothing.

CircleCI Approach
On CircleCI, we can do the following:

- restore_cache:  # restores saved mix cache
      # Read about caching dependencies: https://circleci.com/docs/2.0/caching/
          keys:  # list of cache keys, in decreasing specificity
            - v1-mix-cache-{{ .Branch }}-{{ checksum "mix.lock" }}
            - v1-mix-cache-{{ .Branch }}
            - v1-mix-cache
      - restore_cache:  # restores saved build cache
          keys:
            - v1-build-cache-{{ .Branch }}
            - v1-build-cache
      - run: mix do deps.get, compile  # get updated dependencies & compile them

This can speed up builds dramatically.

Conclusion
I'd love to see Earthly better handle this use case. I'm not sure how to have this play nicely with CACHE and --remote-cache.

For more information, see this Slack thread

[vladaionescu]

We could add support for a flag like --cache-from in earthly (and allow this to be repeated), similar to buildx --cache-from.

earthly --cache-from=someone/img1:latest \
        --cache-from=someone/img2:latest \
        --remote-cache=something/something:latest \
        +my-target

This is already supported by BuildKit, so adding this should not be difficult. It just needs to get to this point in the code: https://github.com/earthly/earthly/blob/main/builder/solver.go#L208.

FWIW, there is already a SAVE IMAGE --cache-from, which could be used as a workaround, which should work theoretically, if placed, for example in the target being called directly. But this isn't very elegant, as there's no image actually being saved.

[alexcb]

I added a relevant CHANGELOG item under d70cd24