Kaniko doesn't cache in Multistage when copying files from a modified stage

[thaituan]

Actual behavior
In multistage, caching when copying file from a stage (which has change, although that change does not effect on copied file) to other stage does not happen.
It seems that this happened after fixing issue #589

Expected behavior
Layers that haven't changed on the previous stage will still be cached on the next stage.

To Reproduce
Steps to reproduce the behavior:

1. See https://github.com/thaituan/kaniko-cache
2. Build docker image with cache on
3. Update scripts in package.json (not change dependencies and devDependencies)
   Example: "test": "echo \"Error: no test specified\" && exit 10" -> "test": "echo \"Error: no test specified\" && exit 20"
4. Re-build docker image with cache on

Additional Information

• Dockerfile: https://github.com/thaituan/kaniko-cache/blob/master/Dockerfile

FROM node:10-slim AS deps
COPY package.json /tmp
RUN cat /tmp/package.json | xargs -0 -i% node -pe 'o=(%);JSON.stringify({dependencies:o.dependencies,devDependencies:o.devDependencies})' > /tmp/package.deps.json

FROM node:10-slim
WORKDIR /app
COPY --from=deps /tmp/package.deps.json ./package.json
COPY ["yarn.lock", "./"]
RUN yarn install

Line 7 is supposed to be cached when only dependencies and devDependencies are changed.
Using the docker command to build works fine, but kaniko not.

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[tejal29]

@thaituan The way kaniko caching works right now, it also adds the cache key for deps when calculating the cache key. You can make a change here

kaniko/pkg/executor/build.go

Line 186 in 03b3391

compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)

and that should fix your issue.

[thaituan]

@tejal29 thank for your responding. I see in line:

kaniko/pkg/executor/build.go

Line 186 in 03b3391

compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)

and line:

kaniko/pkg/executor/build.go

Line 188 in 03b3391

compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)

are the same with 2 different cases. It looks like the key cache of the deps is being generated incorrectly. Is there a way to create a cache key that works the same way as the docker build?

[tejal29]

We need to remove it from both cases i.e.

• when a cached command is used
• when a regular command is used.

Please let me know if you need any more pointers.

[thaituan]

@tejal29
Thank you for the help.
I tried removing this block:

kaniko/pkg/executor/build.go

Lines 184 to 189 in 03b3391

	switch v := command.(type) {
	case *commands.CopyCommand:
	compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)
	case *commands.CachingCopyCommand:
	compositeKey = s.populateCopyCmdCompositeKey(command, v.From(), compositeKey)
	}

and it looks like my problem has been solved. Please check back when you have time.

[HubertBos]

Hello

We have a very similar situation today when a multistage container created a binary on the first stage but use a cached layer on 2nd stage.

[gsaraf]

Not quite the same thing, but related - when using the executor:latest image, if we had the --cache-copy-layers flag, it was using cached versions of files copied from previous stages even when they had changed. When we removed that flag, it no longer reproduced.

[massimeddu-sj]

Hi, now that #2065 is merged, it would be great if this could be solved, so it will be safe to re-enable the cache for copy layers in multistage images.

For example, we have few images that where affected a lot by not be able to use the copy layer cache, increasing the build time by around 100%.

@tejal29 @thaituan any chance to see a fix for this issue?

Thank you!