Proposal: Kaniko should have option to use local layer cache

[cvgw]

Proposal: Local layer cache

Current behavior

Kaniko currently supports only a remote layer cache (a docker registry).

Kaniko does allow for caching of base images within a local cache, but it does not support image layers built by kaniko (or other docker image build mechanisms)

Desired behavior

Kaniko should allow users to specify a local directory to use for caching layers. Kaniko should support both reading and writing from this cache.

Kaniko should support any local filepath (symlink, hardlink, mounted volume, etc.)

Why this is needed

A non-trivial amount of time during a build using Kaniko is spent reading and writing image layers to the remote cache.

Allowing users to specify a local cache would greatly optimize execution time for builds using caching. This would be useful for users who run Kaniko as part of their local development or in a Kubernetes cluster.

What is out of scope

• Warming the local cache with layers from a remote cache
• Using a local and remote cache at the same time

How would we implement this

Updating kaniko to both read and write the tar and metadata for each layer from a local directory would be a reasonable change to the current executor pkg. Users could specify a local layer cache by passing a flag which would be used as an alternative to --cache-repo. Kaniko would only allow users to specify a local cache OR --cache-repo.

[cvgw]

related #919

[cvgw]

cc @tejal29

[cvgw]

Also related #496

[tejal29]

Related to #969

[lllamnyp]

A common pattern we employ is

docker pull <previousimage>
docker build --cache-from <previousimage> -t <newimage> .
docker push

In the context of a local cache, could executor --cache-from /path/to/tarball/of/image.tar be a reality?

[CarlosDomingues]

I'm very happy to use Kaniko as it solves many security issues, but local caching could make my CI/CD pipelines much faster. In addition, cloud providers will often charge for data transfer, those costs could be reduced with this feature.

The CI/CD pattern I use is to have one Multi-Stage Dockerfile with one stage for each step (build, test, publish, etc...), then call Kaniko like that:

/kaniko/executor --cache=true --target "${STAGE}" <...>

When using Docker it will store intermediate steps locally making things very fast, unfortunately that's not possible with Kaniko.

[tejal29]

@caarlos0 Thanks for the feedback. Like you mentioned we only have a remote cache support.
We are relying on contributors to help us out. Local cache would be a great feature addition to Kaniko.

I am happy to help, guide to get this feature delivered

[ghost]

Even im looking for this feature, it would be great if we have this feature soon!

[saltbo]

Any progress on this issue?

[tejal29]

I am happy for anyone in the community to help implement this feature. Unfortunately at this time, I dont have the time and bandwidth to implement this.

[quater]

Given that this has not been addressed so far despite reasonable demand, I wonder what the technical challenges would be to get such capability implemented?

[ChongmingDu]

Is it in the plan?

[acanewby]

I am happy for anyone in the community to help implement this feature. Unfortunately at this time, I dont have the time and bandwidth to implement this.

@tejal29 I would be willing to take a look at this. The value is obvious. Would it be possible to chat offline to get your perspective on what might be involved?

[tejal29]

Thanks @acanewby. Would you be up for writing up a design proposal just explaining

1. interfaces/code flow changes
2. how plan to implement the feature, via flags, tests etc

Thanks

[maxpain]

Any updates?

[benzvan]

I would be very much in favor of this but I'm not sure where to start with making it happen. Did that design proposal get created?

[yasseryehya]

I very much need this too. I am involved with testing the image locally for vulnerabilities and only then push it to jfrog artifactory.

• I first build the image locally with /kaniko/executor --destination=image --tarPath=/workspace/image.tar --context="." --no-push
• then perform the scan over image.tar
• If scan is successful, I can push the image to jfrog artifactory with /kaniko/executor --context "." --dockerfile "Dockerfile" --destination "dst_url/test_img:latest". this involves rebuilding the image.

I can use remote cache with --cache=true and first push the image to another repo and then push it again to the main one, in this case the second build will be fast but this is not what I need as I only push if scan is successful. using local cache will help a lot.

for the time being, is there a way to push the already built image.tar to jfrog artifactory?
thanks

[peternann]

Came here to try to understand why this has not been done.
Leaving none the wiser...
This seems like a really obvious (and not too hard?) optimisation Kaniko should have?

[LucasLundJensen]

This is also something that is a must for us, sadly leaving us unable to use Kaniko.
Watching this issue in case it should be added eventually.