yarn is missing files after container is built with kaniko #1071

drptbl · 2020-02-22T11:20:55Z

Dockerfile with yarn is missing files after container is built with kaniko, tried on 0.17.1 and 0.16.0:

FROM node:10.19.0-buster
ENV YARN_VERSION 1.22.0

# install latest yarn
RUN curl -fSLO --compressed "https://yarnpkg.com/downloads/$YARN_VERSION/yarn-v$YARN_VERSION.tar.gz" \
    && tar -xzf yarn-v$YARN_VERSION.tar.gz -C /opt/ \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarn /usr/local/bin/yarn \
    && ln -snf /opt/yarn-v$YARN_VERSION/bin/yarnpkg /usr/local/bin/yarnpkg \
    && rm yarn-v$YARN_VERSION.tar.gz

.. now if you pull this image and cd /opt/yarn-v1.22.0/ you will notice empty folders without files.

The text was updated successfully, but these errors were encountered:

cvgw · 2020-02-25T17:06:18Z

This should be fixed now, can you give tag a1af057f997316bfb1c4d2d82719d78481a02a79 or debug-a1af057f997316bfb1c4d2d82719d78481a02a79 a try?

drptbl · 2020-02-26T09:32:25Z

@cvgw Thanks for looking in to it 🙏. Re-built from scratch using version above, still failing (it passes with success while building with docker run):

$ executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster-kaniko"
00:22
 INFO[0000] Resolved base name us.gcr.io/.../docker-node:10.19.0-buster-kaniko to us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Resolved base name us.gcr.io/.../docker-node:10.19.0-buster-kaniko to us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Built cross stage deps: map[]                
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] Retrieving image manifest us.gcr.io/.../docker-node:10.19.0-buster-kaniko 
 INFO[0000] cmd: USER                                    
 INFO[0000] Checking for cached layer us.gcr.io/.../docker-e2e/cache:e690c8dfc211bcadd3104f3a7f5792b9eda04a6de7692c5496a896f9d8dc3fba... 
 INFO[0000] No cached layer found for cmd RUN apt-get update 
 INFO[0000] cmd: USER                                    
 INFO[0000] Unpacking rootfs as cmd RUN apt-get update requires it. 
 INFO[0011] Taking snapshot of full filesystem...        
 INFO[0020] Resolving paths                              
 INFO[0022] USER root                                    
 INFO[0022] cmd: USER                                    
 INFO[0022] No files changed in this command, skipping snapshotting. 
 INFO[0022] RUN apt-get update                           
 INFO[0022] cmd: /bin/sh                                 
 INFO[0022] args: [-c apt-get update]                    
 error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory
 ERROR: Job failed: command terminated with exit code 1

Dockerfile: https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

cvgw · 2020-02-26T16:43:13Z

@drptbl which docker image are you using? Are you using one of the tags I suggested? I.E gcr.io/kaniko-project/exectutor:a1af057f997316bfb1c4d2d82719d78481a02a79 or gcr.io/kaniko-project/exectutor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79?

I tested your example Dockerfile with both of those images and it built successfully; perhaps I made an error in testing.

Edit: I see the Dockerfile you just linked is not the same one as the original example. I'll take a look at the new one when I have a chance.

drptbl · 2020-02-26T16:48:29Z

@cvgw I've linked full Dockerfile now and started thread with just short version of it :). I've tested it with gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79, everything was being built with new kaniko, base image too (it relies on official node:10-buster image, it's just re-built with kaniko before and used as base for e2e image, so everything is ran with help of kaniko). Sadly it didn't work. Before it was an error related to yarn, now it fails even faster on apt-get install stage.

drptbl · 2020-02-27T23:17:25Z

@cvgw could you please remove fixed-needs-verification label? Thanks 🤝.

cvgw · 2020-02-27T23:20:41Z

@drptbl can you provide instructions on how to build the base image in your latest example or share an example that depends on a public base image. Thanks

drptbl · 2020-02-27T23:27:18Z

@cvgw

Build image named docker-node:10.19.0-buster using kaniko, with following Dockerfile:
https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755

example command format:

executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster"

Build image named docker-e2e:10.19.0-buster using kaniko, with following Dockerfile (it uses previous image as base, please edit):
https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

That's it. You will get an error while building secondary image (docker-e2e):

$ executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster"
00:23
 INFO[0000] Resolved base name us.gcr.io/edited/docker-node:10.19.0-buster to us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Resolved base name us.gcr.io/edited/docker-node:10.19.0-buster to us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Built cross stage deps: map[]                
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0000] Retrieving image manifest us.gcr.io/edited/docker-node:10.19.0-buster 
 INFO[0001] cmd: USER                                    
 INFO[0001] Checking for cached layer us.gcr.io/edited/docker-e2e/cache:3275a5d30d41e3700c70ac09944d85b9e34291b57c7971839f7dcc3091ecba6b... 
 INFO[0001] No cached layer found for cmd RUN apt-get update 
 INFO[0001] cmd: USER                                    
 INFO[0001] Unpacking rootfs as cmd RUN apt-get update requires it. 
 INFO[0012] Taking snapshot of full filesystem...        
 INFO[0020] Resolving paths                              
 INFO[0022] USER root                                    
 INFO[0022] cmd: USER                                    
 INFO[0022] No files changed in this command, skipping snapshotting. 
 INFO[0022] RUN apt-get update                           
 INFO[0022] cmd: /bin/sh                                 
 INFO[0022] args: [-c apt-get update]                    
 error building image: error building stage: failed to execute command: starting command: fork/exec /bin/sh: no such file or directory
 ERROR: Job failed: command terminated with exit code 1

As you can see I'm always clearing cache while trying things, so it's not a case.

cvgw · 2020-02-28T00:23:20Z

@drptbl I cannot repro with the information you've provided using any of the following images

fb0cfd74649f88a942c0a77c85136b791a9c09d1
debug-a1af057f997316bfb1c4d2d82719d78481a02a79
a1af057f997316bfb1c4d2d82719d78481a02a79

Can you please include the exact docker command you are using to execute the build

Dockerfile base https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755
Dockerfile https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

docker command

docker run \
  -v `pwd`:/workspace \
  $IMAGE
  --context dir:///workspace \
  --dockerfile Dockerfile \
  --destination $DESTINATION \
  --cache=true \
  -v info

drptbl · 2020-02-28T00:30:53Z

@cvgw It didn't work for both commands, with cache and without, tried both. Also didn't work with other combinations (tried adding --cleanup and --single-snapshot).

Build docker-node image with: (assuming that Dockerfile is in root directory)
https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755

executor --cache=true --cache-ttl=9h --context ./ --dockerfile ./Dockerfile --destination "us.gcr.io/edited/docker-node:10.19.0-buster"

Build docker-e2e image with: (assuming that Dockerfile is in another root directory)
https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

executor --cache=true --cache-ttl=9h --context ./ --dockerfile ./Dockerfile --destination "us.gcr.io/edited/docker-e2e:10.19.0-buster"

Keep in mind that it also doesn't work for me if I remove --cache and --cache-ttl.

I'm building it on latest stable Kubernetes (gke) through GitLab using kaniko docker image (gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79), some more details about machine:

 Running with gitlab-runner 12.7.0 (58272c27)
   on gke-gitlab-runner-57d548b8f-bxlw7 3HmdzdD_
Using Kubernetes namespace: gitlab-runner
00:00
 Using Kubernetes executor with image gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79 ...
Waiting for pod gitlab-runner/runner-3hmdzdd-project-12747902-concurrent-0dppww to be running, status is Pending
00:06
 Waiting for pod gitlab-runner/runner-3hmdzdd-project-12747902-concurrent-0dppww to be running, status is Pending
 Running on runner-3hmdzdd-project-12747902-concurrent-0dppww via gke-gitlab-runner-57d548b8f-bxlw7...
.............

GitLab config (this way my repo is mounted by default on root dir, using image directly without docker run):

build:kaniko:
  image: $KANIKO_IMAGE
  stage: build
  tags:
    - gke
  script:
    - executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination "$IMAGE_NAME:$TAG-buster-kaniko"
    - echo "============== BUILT IMAGES =============="
    - echo "1. $IMAGE_NAME:$TAG-buster-kaniko"
  cache: {}
  dependencies: []
  allow_failure: true

@edit adding -v info didn't seem to change log output:
https://gist.github.com/drptbl/cbbb188f62364f331221e045aee28544

cvgw · 2020-02-28T00:37:47Z

@drptbl can you please try it using a local docker daemon rather than gitlab's runner

drptbl · 2020-02-28T00:57:56Z

@cvgw Tested on macOS locally and it worked fine.
How to make it work on GitLab runners? Any ideas? 🙏

Some dependencies are missing? docker works fine :(.

Thanks for helping me out and digging in to it.

cvgw · 2020-02-28T01:03:48Z

Thanks for testing @drptbl. I've never used gitlab before but I'm setting an account up now to debug this.

drptbl · 2020-02-28T01:05:30Z

@cvgw I can see how much this tool has reduced ours CI build time, even if output builds aren't usable (yet). Happy to help for such a great project :). Ask me anything.

cvgw · 2020-02-28T16:25:43Z

@drptbl I wasn't able to repro this on Gitlab either; I'm guessing something is different about how I set things up.

My runner is installed on a GKE cluster
I built the base image using docker and pushed it to gitlab registry

Dockerfile base https://gist.github.com/drptbl/3441afdf9b5821d34b8ba399c011d755
Dockerfile https://gist.github.com/drptbl/92b67ccffa2c14bd78b7100e9d777a24

.gitlab-ci.yml

build:kaniko:
  image: $KANIKO_IMAGE
  stage: build
  tags:
    - gke
  script:
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - executor --cache=true --cache-ttl=9h --context $CI_PROJECT_DIR/kaniko --dockerfile $CI_PROJECT_DIR/kaniko/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG
    - echo "============== BUILT IMAGES =============="
    - echo "1. $IMAGE_NAME:$TAG-buster-kaniko"
  cache: {}
  dependencies: []
  allow_failure: true

kaniko image
gcr.io/kaniko-project/executor:debug-a1af057f997316bfb1c4d2d82719d78481a02a79

drptbl · 2020-02-28T16:27:24Z

@cvgw just to confirm, I'm building base image with kaniko and then e2e image with kaniko. You wrote that you built base with docker. Did you mean kaniko or docker? Maybe the case is that base has to be built with kaniko too?

Flow:

1. build base image (docker-node) with kaniko and push it to registry
2. reference it in docker-e2e dockerfile
3. build docker-e2e image with kaniko
^ all of these done on gitlab runners running on kubernetes (which is running on gke) through gitlab.com

cvgw · 2020-02-28T16:36:23Z

@cvgw just to confirm, I'm building base image with kaniko and then e2e image with kaniko. You wrote that you built base with docker. Did you mean kaniko or docker? Maybe the case is that base has to be built with kaniko too?

I indeed built my base image with docker; I'll try building it with kaniko and report back

drptbl · 2020-03-06T21:55:09Z

@cvgw tried v0.18.0-debug, same error.

drptbl · 2020-03-20T18:38:39Z

@cvgw tried v0.19.0-debug, same error.

Frederik-Baetens · 2020-04-25T15:04:14Z

I also have this issue on gitlab runners, with a dockerfile that uses npm to build a project and my /usr/src/app/dist/ went missing when building with kaniko. i switched back to dind for now.

My docker file:

FROM node:12.13-alpine AS builder
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build:raw

FROM node:12.13-alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install --unsafe-perm --only=production
COPY --from=builder ./dist ./dist
ENV NODE_ENV production
# Need to be set by deployment
ENV PROVIDER ''
EXPOSE 3000
CMD ["npm", "run start:raw"]

Edit: my issue is probably related to #552 and #1206

drptbl · 2020-04-25T15:25:57Z

I also have this issue on gitlab runners, with a dockerfile that uses npm to build a project and my /usr/src/app/dist/ went missing when building with kaniko. i switched back to dind for now.

Yup, there is no other way. I hope it will be fixed soon, kaniko is a great project 🎉.

jmuleiro · 2021-08-13T15:58:11Z

Hello there!
I know this issue is rather old, but as it is still open, I'm sharing my experience with it.

I'm running Kaniko in Kubernetes as a temporary pod.

The Dockerfile is rather large, but nothing seemed to help the fact that all files were deleted once Kaniko pushed the built image to the destination. My solution came from changing the executor's build options.

Pod args:

# ...
    args:
      - "--context=<context>"
      - "--dockerfile=./docker/Dockerfile-stage"
      - "--destination=<gcr-destination>"
      - "--ignore-path=./node_modules"
      - "--ignore-path=./k8s"
      - "--ignore-path=./envs"
      - "--ignore-path=./scripts"
      - "--snapshotMode=time"
      - "--cache=true"
      - "--cache-ttl=9h"
# ..

I used to have the single-snapshot option on - this seemed to be the issue somehow. I removed it and now it works as expected.

mecampbellsoup · 2023-02-13T17:17:12Z

Perhaps related to #2065 ?

cvgw added area/filesystems For all bugs related to kaniko container filesystems (mounting issues etc) fixed-needs-verfication kind/bug Something isn't working labels Feb 25, 2020

cvgw self-assigned this Feb 25, 2020

cvgw added more-information-needed and removed fixed-needs-verfication labels Feb 27, 2020

cvgw removed the more-information-needed label Feb 28, 2020

tejal29 added the priority/p2 High impact feature/bug. Will get a lot of users happy label May 7, 2020

tejal29 added this to the Release v1.1.0 milestone May 7, 2020

aaron-prindle unassigned cvgw Jul 6, 2023

aaron-prindle added issue/missing-files differs-from-docker works-with-docker uncategorized labels Jul 13, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

yarn is missing files after container is built with kaniko #1071

yarn is missing files after container is built with kaniko #1071

drptbl commented Feb 22, 2020 •

edited

Loading

cvgw commented Feb 25, 2020

drptbl commented Feb 26, 2020 •

edited

Loading

cvgw commented Feb 26, 2020 •

edited

Loading

drptbl commented Feb 26, 2020 •

edited

Loading

drptbl commented Feb 27, 2020

cvgw commented Feb 27, 2020

drptbl commented Feb 27, 2020 •

edited

Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 •

edited

Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 •

edited

Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 •

edited

Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 •

edited

Loading

cvgw commented Feb 28, 2020 •

edited

Loading

drptbl commented Mar 6, 2020

drptbl commented Mar 20, 2020

Frederik-Baetens commented Apr 25, 2020 •

edited

Loading

drptbl commented Apr 25, 2020

jmuleiro commented Aug 13, 2021 •

edited

Loading

mecampbellsoup commented Feb 13, 2023

yarn is missing files after container is built with kaniko #1071

yarn is missing files after container is built with kaniko #1071

Comments

drptbl commented Feb 22, 2020 • edited Loading

cvgw commented Feb 25, 2020

drptbl commented Feb 26, 2020 • edited Loading

cvgw commented Feb 26, 2020 • edited Loading

drptbl commented Feb 26, 2020 • edited Loading

drptbl commented Feb 27, 2020

cvgw commented Feb 27, 2020

drptbl commented Feb 27, 2020 • edited Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 • edited Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 • edited Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 • edited Loading

cvgw commented Feb 28, 2020

drptbl commented Feb 28, 2020 • edited Loading

cvgw commented Feb 28, 2020 • edited Loading

drptbl commented Mar 6, 2020

drptbl commented Mar 20, 2020

Frederik-Baetens commented Apr 25, 2020 • edited Loading

drptbl commented Apr 25, 2020

jmuleiro commented Aug 13, 2021 • edited Loading

mecampbellsoup commented Feb 13, 2023

drptbl commented Feb 22, 2020 •

edited

Loading

drptbl commented Feb 26, 2020 •

edited

Loading

cvgw commented Feb 26, 2020 •

edited

Loading

drptbl commented Feb 26, 2020 •

edited

Loading

drptbl commented Feb 27, 2020 •

edited

Loading

drptbl commented Feb 28, 2020 •

edited

Loading

drptbl commented Feb 28, 2020 •

edited

Loading

drptbl commented Feb 28, 2020 •

edited

Loading

drptbl commented Feb 28, 2020 •

edited

Loading

cvgw commented Feb 28, 2020 •

edited

Loading

Frederik-Baetens commented Apr 25, 2020 •

edited

Loading

jmuleiro commented Aug 13, 2021 •

edited

Loading