Caching with the combination of COPY --from with RUN #1600
Comments
|
We faced exactly the same issue. So instead of COPY --from=builder /release/ $APP_HOME/
RUN chown -R $APP_USER:$APP_GROUP $APP_HOME/we run COPY --chown=$APP_USER:$APP_GROUP --from=builder /release/ $APP_HOME/ |
|
I experienced a similar issue when downloading a file using I'm not sure if it's possible to use |
aaron-prindle
added
area/caching
|
Looking into the codebase, it seems that the invalidation of COPY command needs to be added at: pkg/commands/copy.go#L152 where it previously doesn't invalidate the COPY command unless a cache key miss is hit. |
|
I had a chance to learn from @aaron-prindle that from previous design discussion and decisions made, there are currently only supports for #3018 is raised as a new feature request and #3019 for documentation clarification. |
|
I will go ahead and close this issue for now and please redirect to #3018 Please feel free to reopen if necessary. |
|
I think this issue should not have been closed. Not invalidating the cache for the subsequent It is a bug regardless of whether a feature like #3018 exists that allows to work around the bug in some cases. |
Thanks @felixhuttmann for chiming in. I reopened this for more discussion. From the previous use cases, the bug of invalidated cache being used happens where the file might not previously have the expected permissions. Could you confirm if the case still persists without |
|
@felixhuttmann would appreciate it if you don't mind following up on this with more evidence? |
|
@JeromeJu After trying again, I was not able to reproduce this. Feel free to close this issue again. Sorry for the noise. I was confused, because you wrote above
but I did not see that any code there recently changed, and this issue also does not link to any MR where a fix was performed. Perhaps this issue was fixed as part of something else in the meantime. |
|
I think we just ran into this issue: Stripped down Dockerfile: FROM buildpack-deps:bookworm as base
RUN useradd --user-group --create-home --shell /bin/bash user
SHELL ["/bin/bash", "-c"]
RUN mkdir -p /workdir
RUN mkdir -p /workdir && chown user:user /workdir
WORKDIR /workdir
USER user
# … (install NodeJS/npm/pnpm here)
COPY --chown=user:user ./pnpm-lock.yaml .
# ------------------------------------------------
FROM base AS e2e_browser_dependencies_prepare
WORKDIR /workdir
USER user
COPY --chown=user:user pnpm-lock.yaml .
RUN grep --extended-regexp ' playwright: \d+\.\d+\.\d+' pnpm-lock.yaml | cut -d ':' -f 2 | cut -d ' ' -f 2 > playwright_version
# ------------------------------------------------
FROM base AS e2e_browser_dependencies
WORKDIR /workdir
USER user
COPY --from=e2e_browser_dependencies_prepare /workdir/playwright_version .
# The following two RUN commands get retrieved from cache even when pnpm-lock.yaml and playwright_version have changed!
RUN VERSION_FROM_PACKAGE_JSON=$(cat playwright_version) bash -c 'npm install "playwright@${VERSION_FROM_PACKAGE_JSON}"'
RUN npx playwright install chromiumThe Kaniko logs in our Gitlab job show that |
Thanks @codethief for your inputs. From your Dockerfile, it looks like Copy -chown has worked as expected, which does not retrieve from cache. Could you confirm if the issue was only related with |
Indeed, good point.
I'm not sure I'm following. The |
Thanks for confirming that the COPY -chown worked as expected. By that I was trying to decouple the issues here in order to find the root cause for your use case given that this issue was raised under the circumstance where IIUC, in your use case would there be a workaround if you move |
|
Thanks for looking into this, @JeromeJu – it's highly appreciated!
I see. Yeah, it's probably not only
Unfortunately, this wouldn't work. I need the right NPM package to be installed in the last stage in order for the very last command |
Actual behavior
kaniko uses the cached version of a
RUN chown ... /app/buildcommand although the copied source has changes in the previous commandCOPY --from=builder /app/build /app/buildand it shouldn't be using caches starting from this command.As a result, in the image we have the changed copied source that have been overrided by the cache from the
RUNcmd.This behaviour has been confirmed using the tool dive.
Expected behavior
As it is stated in the kaniko README.md:
The command
COPY --from=builder /app/build /app/buildhas detected changes.Thus, the cache must have been invalidated for the subsequent commands such as
RUN chown ... /app/buildTo Reproduce
Steps to reproduce the behavior:
COPY --from=builder /app/build /app/buildcmd of your build app followed by aRUN chown ... /app/buildcmd)--cache=trueto create caches.COPY --frombut it will be overrided by the cache ofRUN chowncmd.Additional Information
Information about the build of the docker image:
We use gitlab-runner to launch kaniko.
Our application is a front app (vue.js/typescript). We use webpack (v4) to build our app.
Version of kaniko: 1.5.1
Kaniko's image: gcr.io/kaniko-project/executor:debug
Flags used: --cache=true --context PROJECT_DIR --dockerfile PATH_DOCKERFILE --destination REGISTRY:TAG
The Docker Image is pushed to our Gitlab Container Registry.
The command:
/kaniko/executor --cache=true --context PROJECT_DIR --dockerfile PATH_DOCKERFILE --destination REGISTRY:TAGDockerfile:
Our Dockerfile with 2 stages:
Output:
Here is the kaniko's output of the strange behavior when we build the image:
Our Investigation:
My team and I investigated this issue to provide more information.
Step 1: Build a docker image with cache
On our pipeline, we first build the docker image with our current front app.
Then, we change the wording in our application, ex: "delete" => "deleteA".
A new docker image is built using for some parts the cache.
Result: the wording "deleteA" isn't applied when deploying our app.
The logs are above in the Output part.
Step 2: Extraction of the build app folder "dist/"
In the image produced from step 1, we extracted the files in the folder "dist" (result of the build using webpack).
The structure of the folder was:
We noticed that there were 2 app..css and app..jss in
static/css&static/js. It was not normal.We started to look inside the 2 app..js in
static/js.And we found something interesting, the two app..js were 2 versions of our front app, one with the wording "delete" and the other with the wording "deleteA".
We then investigated the index.html, and understood that the src used in this file were the app.*.js with the "delete" (not the one we wanted).
Step 3: Using dive to see what was happening in our image
We then started to explore our image with dive.
We saw that all the layers in the 1st stage was normal.
In the 2nd stage, we saw that the cmd
COPY --from=node-builder /app/dist /usr/share/nginx/htmldid his job correctly by not using the cache.Then in the layer of the cmd
RUN chown -R 1001:1001 /usr/share/nginx/html, we saw that some files were been overrided (ex: index.html, runtime*.js, vendors*.js) and that the app.*.js (the previous version of our app with the wording "delete") was added in our build directory.Conclusion: the
RUN chown -R 1001:1001 /usr/share/nginx/htmlcmd was using the cache.What we did
After that we discovered the cmd
COPY --chown, we tried to use it but it didn't work as stated in this issue #1456 (I added a comment in this issue too).Triage Notes for the Maintainers
--cacheflagThe text was updated successfully, but these errors were encountered: