Proposal: optionally export cache metadata in the image config

[tonistiigi]

BuildKit supports exporting build cache and pulling back the parts that it knows will produce matches. There are currently 2 export modes: max that will export all the layers of all intermediate steps and min that will only export layers for the resulting image(that are presumably exported anyway) and only metadata for all the intermediate steps. Having only metadata for the intermediate steps is already very beneficial in lots of cases and can still effectively cache multi-stage builds.

Currently, the UI for exporting cache in docker does not exist and because the cache format uses manifest lists and docker's buildkit integration is not using containerd it is unlikely that it can be added for 19.03.

As an alternative, we could support sharing cache inside the image config that we should be able to easily use in docker as well.

Basically, we would do the same thing we do with current min cache mode but instead of pushing the metadata+final layers to a separate reference with a manifest list we would inline the metadata directly into image config. The image config already has the layer information.

{
   ....
   "moby.buildkit.buildcache": <base64>
}

Then this image can be used directly in the docker build --cache-from.

Note that one of the problems with the implementation is that while currently, the image config(digest) is reproducible in case of a cache match, the build cache atm is not. This is because the local sources use random IDs for their definition based cache keys. So either this issue needs to be solved and these cache keys removed from the exported selection (eg. marking these keys as not for exporting) or we need to accept new digests on every build(old docker behavior).

This should also be optional and cache should only be produced when it is requested.

[AkihiroSuda]

SGTM