Do not recompute layers retrieved from cache

[cvgw]

Description
Updates kaniko to reuse cached layers rather than recreating the layer.

Previously kaniko would retrieve a layer from the cache, unpack the tar, snapshot the file system, and then repack the tar, recreate the layer. This caused unneeded work repacking the ta. It also resulted in layer and image digests that would change every build due to the tar being repacked.

This change updates kaniko so that the new behavior is

• Retrieve layer from cache
• Unpack tar
• Snapshot filesystem
• Reuse layer retrieved from cache

Note that for the moment it is only implemented for RUN and COPY commands.

This eliminates the extra steps of repacking the tar. It also results in layer and image digest which do not change for cached layers.

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

• Includes unit tests
• Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

• The code flow looks good.
• Unit tests and or integration tests added.

Release Notes

- Improve performance when caching is enabled
- Improve caching behavior for FROM commands which reference a previous stage

[tejal29]

lgtm!

[cvgw]

This actually will fix the specific case in #682 but not the underlying issue. PR #891 should be a more complete fix.

This change may still be worth doing to remove the extra work of recreating the tar.

[tejal29]

@cvgw Would be able to pick this up for release tomorrow? if not i will rebase and merge.

[cvgw]

@tejal29 do we still want to make this change? I'm not sure what the motivation is now that we've made all the caching fixes. I suppose there could be a performance win here.

WDYT?

[tejal29]

Since you have spent cycles on it, we can get this in. Upto you!

[cvgw]

This has a LOT of conflicts with master and some substantial overlap with #973

I'm going to wait until #973 is merged and then cherry-pick the changes over

[raijinsetsu]

I believe this PR and/or #715 are contributing to significant time in our builds. Is there an ETA when these changes may be available?
Note: #973 has been merged as of 2 weeks ago.

[tejal29]

We should look into how we can measure performance increase due to this PR.
One think we can do is,

1. Monitor build time decrease for GCB (internally)

[raijinsetsu]

@tejal29
Based on the time I am seeing in my build spent re-archiving and sending the archive to Cloud Storage, my build time would be reduced by about 10 minutes in the "cache hit" scenario, or 33%. If you think it would help, I could try to build and use master to give you actual numbers.

Note: early in my Dockerfile is a "RUN npx lerna bootstrap --ci" which installs all the dependencies for the mono-repository. Installing the dependencies takes about 10 minutes, which includes building 15+ local modules. The subsequent archiving takes about 10 minutes. As the dependencies do not change often, having the cache work correctly would trim about 20 minutes from most builds.

[raijinsetsu]

Just for giggles, I used the "gcr.io/kaniko-project/executor:debug-8d9e6b8ea54274f73517f11c113c13cd03d26349" image which appears to have this change.
My build went from 20 minutes to 2 minutes. 👍