Missing files in node_modules when using --use-new-run=true

[fabiotamagno]

Actual behavior
With --use-new-run=true, files that exist on the first kaniko run under ./node_modules/ folder, are not available when using the cache.

This happens with the same pattern on every run, and the same files are missing.

The following is from a run with caches:

 INFO[0053] Running: [/bin/sh -c ls -lah ./node_modules/react-app-rewired/bin/] 
 total 8.0K
 drwxr-xr-x 2 root root 4.0K Jun 14 00:57 .
 drwxr-xr-x 5 root root 4.0K Jun 14 00:57 ..
 INFO[0053] No files changed in this command, skipping snapshotting. 

(above there should be at least index.js)

 INFO[0054] Running: [/bin/sh -c ls -lah ./node_modules/.bin/] 
 total 40K
 drwxr-xr-x    2 root root 4.0K Jun 14 00:57 .
 drwxr-xr-x 1140 root root  36K Jun 14 00:57 ..
 lrwxrwxrwx    1 root root   34 Jun 14 00:57 detect -> ../detect-port-alt/bin/detect-port
 lrwxrwxrwx    1 root root   34 Jun 14 00:57 detect-port -> ../detect-port-alt/bin/detect-port
 lrwxrwxrwx    1 root root   16 Jun 14 00:57 sha.js -> ../sha.js/bin.js
 INFO[0055] No files changed in this command, skipping snapshotting. 

(above there should be dozens of symlinks)

On first build, without caches available, the following appears a couple times:

 INFO[0133] Taking snapshot of files...                  
 INFO[0134] Pushing layer gcr.io/******/frontend/cache:abdcef to cache now 
 WARN[0140] error uploading layer to cache: open : no such file or directory

Expected behavior
The same files should be present on every run, even from cache.

 INFO[0072] Running: [/bin/sh -c ls -lah ./node_modules/react-app-rewired/bin/] 
 total 16K
 drwxr-xr-x 2 root root 4.0K Jun 14 01:08 .
 drwxr-xr-x 5 root root 4.0K Jun 14 01:08 ..
 -rwxr-xr-x 1 root root 1.6K Oct 26  1985 index.js
 -rw-r--r-- 1 root root 1.2K Oct 26  1985 jest.js
 INFO[0073] No files changed in this command, skipping snapshotting. 

 INFO[0074] Running: [/bin/sh -c ls -lah ./node_modules/.bin/] 
 total 40K
 drwxr-xr-x    2 root root 4.0K Jun 14 01:08 .
 drwxr-xr-x 1140 root root  36K Jun 14 01:08 ..
 lrwxrwxrwx    1 root root   18 Jun 14 01:08 acorn -> ../acorn/bin/acorn
 lrwxrwxrwx    1 root root   26 Jun 14 01:08 ansi-html -> ../ansi-html/bin/ansi-html
 lrwxrwxrwx    1 root root   19 Jun 14 01:08 atob -> ../atob/bin/atob.js
 lrwxrwxrwx    1 root root   32 Jun 14 01:08 autoprefixer -> ../autoprefixer/bin/autoprefixer
 lrwxrwxrwx    1 root root   25 Jun 14 01:08 babylon -> ../babylon/bin/babylon.js
  lrwxrwxrwx    1 root root   22 Jun 14 01:08 browserslist -> ../browserslist/cli.js
 lrwxrwxrwx    1 root root   26 Jun 14 01:08 css-blank-pseudo -> ../css-blank-pseudo/cli.js
 lrwxrwxrwx    1 root root   24 Jun 14 01:08 css-has-pseudo -> ../css-has-pseudo/cli.js
 lrwxrwxrwx    1 root root   34 Jun 14 01:08 css-prefers-color-scheme -> ../css-prefers-color-scheme/cli.js
 lrwxrwxrwx    1 root root   20 Jun 14 01:08 cssesc -> ../cssesc/bin/cssesc
 lrwxrwxrwx    1 root root   34 Jun 14 01:08 detect -> ../detect-port-alt/bin/detect-port
 lrwxrwxrwx    1 root root   34 Jun 14 01:08 detect-port -> ../detect-port-alt/bin/detect-port
 lrwxrwxrwx    1 root root   15 Jun 14 01:08 errno -> ../errno/cli.js
 lrwxrwxrwx    1 root root   29 Jun 14 01:08 escodegen -> ../escodegen/bin/escodegen.js
 lrwxrwxrwx    1 root root   30 Jun 14 01:08 esgenerate -> ../escodegen/bin/esgenerate.js
 lrwxrwxrwx    1 root root   23 Jun 14 01:08 eslint -> ../eslint/bin/eslint.js

 [...]
 INFO[0075] No files changed in this command, skipping snapshotting. 

To Reproduce
Steps to reproduce the behavior:

1. Create a base "create-react-app" project

npx create-react-app my-app
cd my-app

2. Put the Dockerfile at the root
3. Build the image with the Dockerfile in Kaniko
4. Make a change to the JS code
5. Rebuild the image with the Dockerfile in Kaniko

Additional Information

• Dockerfile

# build environment
FROM node:13.7-stretch as builder
RUN mkdir /app
WORKDIR /app

COPY ./package-lock.json /app/package-lock.json
COPY ./package.json /app/package.json
RUN npm ci
COPY ./. /app/

RUN ls -lah ./node_modules/.bin/
RUN npm run build

# production environment
FROM nginx:1.15-alpine
COPY --from=builder /app/build /usr/share/nginx/html

CMD ["nginx", "-g", "daemon off;"]

• Build Context
  • Create-React-App directory
  • Gitlab-CI Runner on GKE
  • Using Kaniko Performance Testing.  #1305 suggestions
• Kaniko Image (fully qualified with digest)
  gcr.io/kaniko-project/executor@sha256:9f17434aa98dc64d98350dee516927bebae1a1d40a9cac305d1d5e5864e4c850 debug-perf

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[tejal29]

Thanks @fabiotamagno for the feedback.
I will look into this.

[msembinelli]

Confirming that I also see this issue! Would really like to see this fixed as the new run command was SO much faster, however all my projects are node...

[tejal29]

@msembinelli i did not get time to investigate this.

Did you try using the new --snapshotMode=redo to see if that improved your timings?

[tejal29]

hey @fabiotamagno, i had a small bug in the code #1379
Can you try your build using this image?

gcr.io/kaniko-project/executor:latest-1317
gcr.io/kaniko-project/executor:debug-1317

I verified this image here #1317 (comment)

[tejal29]

closing as dupe of #1317

[retpolanne]

@tejal29 didn't work for me, maybe the issue is different?

I also have a build that includes node_modules and it's missing when I build using kaniko (with the image you provided).

[robross0606]

I am also seeing this and baffled. Our dockerfile contains a RUN npm ci command which executes fine and we can see the log entries during the build. However, the resulting image does not contain the node_modules directory at all. We also have no .dockerignore file in play here.