when replacing a folder by symlink : i get corrupt images when pulling : failed to mknod("somefile", S_IFCHR, 0): no such file or directory #2308
Comments
|
Minimal reproducer: Dockerfile FROM busybox
RUN mkdir /a /b /c && echo a > /a/a
RUN rm -Rfv /a && ln -sf /b /aBuild with: podman run --rm -ti --volume $PWD:/workspace gcr.io/kaniko-project/executor:v1.9.1-debug --tar-path /workspace/out.tar --no-push --destination dummy:latestThe resulting tar file cannot be loaded into podman/docker: |
|
Haa thanks :) |
|
Bisecting yields
as the culprit, which sounds related. |
|
Hmm not sure about, I refactored the strange whiteout files once to be clearer, there was lots of old stuff laying around, but I think that commit came afterwards? My refactor was about to track correctly deleted and new files for each layer and act on these two lists later in the process when the tar for the layer is created. #2066 Info: the line |
|
any update on the above ? this is becoming a real issue :( |
|
Oopsie. I think I know what the problem is and I'll prepare a fix |
If a non-empty directory gets replaced with a link, the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes GoogleContainerTools#2308
|
@dannygueta @TobiX this branch fixes the minimal testcase: https://github.com/andreasf/kaniko/tree/fix_2308 Can you try it with some other Dockerfiles that triggered the issue? If you cannot build kaniko images yourself, I pushed a version to |
|
Testing the above Docker file: �[36mINFO�[0m[0001] RUN mkdir /a /b /c && echo a > /a/a
�[36mINFO�[0m[0001] Initializing snapshotter ...
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[]
�[37mDEBU�[0m[0001] Adding to layer: [/bin/strings /bin/id /bin/ftpput /bin/lo
�[37mDEBU�[0m[0001] Deleting in layer: map[]
�[36mINFO�[0m[0001] Cmd: /bin/sh
�[36mINFO�[0m[0001] Args: [-c mkdir /a /b /c && echo a > /a/a]
�[36mINFO�[0m[0001] Running: [/bin/sh -c mkdir /a /b /c && echo a > /a/a]
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[/:d2c5427ffdb4fe186a7456ccfb
�[37mDEBU�[0m[0001] Adding to layer: [/ /c /a /a/a /b]
�[37mDEBU�[0m[0001] Deleting in layer: map[]
�[36mINFO�[0m[0001] RUN rm -Rfv /a && ln -sf /b /a
�[36mINFO�[0m[0001] Cmd: /bin/sh
�[36mINFO�[0m[0001] Args: [-c rm -Rfv /a && ln -sf /b /a]
�[36mINFO�[0m[0001] Running: [/bin/sh -c rm -Rfv /a && ln -sf /b /a]
removed '/a/a'
removed directory: '/a'
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[/:da7bc1a25453bea1264515af13
�[37mDEBU�[0m[0001] Adding to layer: [/ /a /b]
�[37mDEBU�[0m[0001] Deleting in layer: map[/a/a:{}]
�[37mDEBU�[0m[0001] Mapping stage idx 0 to digest sha256:96140bb5279053f61b690
�[37mDEBU�[0m[0001] Mapping digest sha256:96140bb5279053f61b6900ab4c7bb7b180a3
�[36mINFO�[0m[0002] Skipping push to container registry due to --no-push flag
at �[36mINFO�[0m[0001] RUN mkdir /a /b /c && echo a > /a/a
�[36mINFO�[0m[0001] Initializing snapshotter ...
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[]
�[37mDEBU�[0m[0001] Adding to layer: [/bin/sed /bin/kill /bin/dmesg /bin/stat
�[37mDEBU�[0m[0001] Deleting in layer: map[]
�[36mINFO�[0m[0001] Cmd: /bin/sh
�[36mINFO�[0m[0001] Args: [-c mkdir /a /b /c && echo a > /a/a]
�[36mINFO�[0m[0001] Running: [/bin/sh -c mkdir /a /b /c && echo a > /a/a]
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[/:57448699613d56efe35c1627a
�[37mDEBU�[0m[0001] Adding to layer: [/b / /c /a /a/a]
�[37mDEBU�[0m[0001] Deleting in layer: map[]
�[36mINFO�[0m[0001] RUN rm -Rfv /a && ln -sf /b /a
�[36mINFO�[0m[0001] Cmd: /bin/sh
�[36mINFO�[0m[0001] Args: [-c rm -Rfv /a && ln -sf /b /a]
�[36mINFO�[0m[0001] Running: [/bin/sh -c rm -Rfv /a && ln -sf /b /a]
removed '/a/a'
removed directory: '/a'
�[36mINFO�[0m[0001] Taking snapshot of full filesystem...
�[37mDEBU�[0m[0001] Current image filesystem: map[/:70247bf83a3da8ea4d48b2455
�[37mDEBU�[0m[0002] Adding to layer: [/b / /a]
�[37mDEBU�[0m[0002] Deleting in layer: map[/a/a:{}]
�[37mDEBU�[0m[0002] Mapping stage idx 0 to digest sha256:1cc290ca89c3ea8e9a55
�[37mDEBU�[0m[0002] Mapping digest sha256:1cc290ca89c3ea8e9a55e34ff7533daf160
�[36mINFO�[0m[0002] Skipping push to container registry due to --no-push flagoutput seems identical and the error seems to be in We have to look close to this function why it was coded/changed like it was. Further investigation needed. |
|
When I print the offending last layer with mkdir layers && tar -xf out.tar && cd layers
tar -tvf last-layer.tar.gzI get for ---------- 0/0 0 1970-01-01 01:00 a/.wh.a
drwxr-xr-x 0/0 0 2023-06-19 22:35 /
lrwxrwxrwx 0/0 0 2023-06-19 22:35 a -> /b
drwxr-xr-x 0/0 0 2023-06-19 22:35 b/and for drwxr-xr-x 0/0 0 2023-06-19 22:43 /
lrwxrwxrwx 0/0 0 2023-06-19 22:43 a -> /b
---------- 0/0 0 1970-01-01 01:00 a/.wh.a
drwxr-xr-x 0/0 0 2023-06-19 22:43 b/which seems to crash on load. Anybody knows the difference? I suppose there is a relation ship between deleting and creating symlinks.
I try to make a fix for reviewing. I am not 100% sure. |
|
Building the Dockerfile above with kaniko 1.8.1, I am getting the following third layer: The layer contains a whiteout file for With kaniko 1.9.1, the order of files in the tar file is different, because there was a problem running images in Cloud Foundry when a file was written before its parent directory.
Docker 24.0.2 only writes the symlink itself to the last layer. It doesn't add a whiteout file for My proposed fix is to not write whiteout files if a directory is replaced with a symlink: This is now relatively close to Docker's output. Kaniko also adds / and the link target, which should not be necessary. I don't think reverting to the old code is a good alternative. It would lead to issues extracting images in Cloud Foundry, and as we saw further up |
|
@andreasf : Ok, I guess my PR still solves the issue. But you are probably right: So is that correct?: What happens if the last layer contains and no whitout for Also: I am wondering why there is no delete for dir |
|
I just realized that the issue is not specific to symlinks. When the directory
Yes, the link is new, so I assume it should be in "adds" (I did not try to fully understand the LayeredMap implementation)
Yes |
|
Test case with file replacing directory: FROM busybox
RUN mkdir /a /b /c && echo a > /a/a
RUN rm -Rfv /a && echo "foo" > /a |
…es or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes GoogleContainerTools#2308
…es or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes GoogleContainerTools#2308
|
@andreasf |
|
@ptempier the problem and the fix are actually the same in both cases. The fix will not affect the ability to use symlinks, it is more about how file system changes are represented in layer tar files, something that users shouldn't have to worry about (assuming everything works): In this case, Kaniko would also add a whiteout file |
|
Very good explanation.I have to check again why the symlink does not appear in the adds struct. Weird.Von meinem iPhone gesendetAm 21.06.2023 um 11:51 schrieb Andreas Fleig ***@***.***>:
@ptempier the problem and the fix are actually the same in both cases. The fix will not affect the ability to use symlinks, it is more about how file system changes are represented in layer tar files, something that users shouldn't have to worry about (assuming everything works):
Layer 1: two directories and a file in foo
/
├─ foo/
│ ├─ bar
├─ baz/
Layer 2: directory foo (which had a file in it) is replaced by a file or a symlink
/
├─ foo
├─ baz/
In this case, Kaniko would also add a whiteout file foo/.wh.bar to layer 2's tar file, because the file foo/bar was present in layer 1 but on layer 2. But because in the same tar file, foo exists and is not a directory anymore, there should not be any files with names like foo/.wh.bar which expect that foo is a directory. This contradiction was why the tar files could not be extracted.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi I resurrected a pipeline that i converted to dind dues to this issue, back to kaniko to test the fixes. Maybe i tested incorrectly, but with both theses versions : The build and pull seems to work. (seems like an improvement to earlier versions, where i couldn't pull) INFO[0013] RUN mkdir -p /data/log/ && mv /var/log/ /var/log.orig && ln -sf /data/log /var/log or with another image : INFO[0001] RUN dnf -y install python-dns python-inotify fail2ban && systemctl enable fail2ban |
…es or links If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes GoogleContainerTools#2308
|
Is it possible that your cache still contains invalid layers? I tried to reproduce your second case but it works for me, but of course I don't have a cache with old data: |
|
@andreasf : Sorry I was mistaken, everything works fine, |
aaron-prindle
added
area/symlinks
area/filesystems
aaron-prindle
added
kind/bug
|
I specified a different registry for cache and it works with the cache as well. |
|
@andreasf Found some time to test your fixed image ( |
…es or links (#2590) If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes #2308
…es or links (GoogleContainerTools#2590) If a non-empty directory gets replaced with something other than a directory (e.g. file or symlink), the files in that directory also get deleted. However, there should not be any whiteout files for them in the layer. If there were, the resulting tar file would not be extractable. Fixes GoogleContainerTools#2308
Actual behavior
When replacing a folder by symlink i get corrupt images , docker pull will fail with :
Failed to register layer: Error processing tar file(exit status 1): failed to mknod("/etc/fail2ban/action.d", S_IFCHR, 0): no such file or directory
Expected behavior
Be able to pull the image
To Reproduce
The error doesn't seems t happen with every files/folder.
I tried disabling all caches to be sure
The build/pull works when using the dockerfile directly with docker build command
The build works with kaniko but appears corrupt as i can t pull it
Additional Information
FROM quay.io/centos/centos:stream9
RUN dnf -y update && dnf -y install systemd
RUN /usr/bin/systemctl mask systemd-logind && /usr/bin/systemctl mask systemd-hostnamed && /usr/bin/systemctl mask dbus-
RUN ln -s /data/git_conf/conf/systemd/journald.conf.d /etc/systemd/journald.conf.d
RUN dnf install -y rsyslog && mv -f /etc/rsyslog.d /etc/rsyslog.d.orig && ln -s /data/git_conf/conf/rsyslog/rsyslog.d /etc/rsyslog.d
RUN systemctl enable rsyslog
RUN dnf -y install procps net-tools
STOPSIGNAL SIGRTMIN+3
CMD [ "/sbin/init" ]
This one build and pull fine even with the the broken symlink that will lead to a mounted volume set in docker compose
FROM previousdockerimage
RUN dnf -y update && dnf install -y epel-release && dnf -y install fail2ban && systemctl enable fail2ban && dnf clean all
RUN rm -rf /etc/fail2ban && ln -s /data/git_conf/conf/fail2ban /etc/fail2ban
STOPSIGNAL SIGRTMIN+3
EXPOSE 22
CMD [ "/sbin/init" ]
When pulling this one errors with
Error processing tar file(exit status 1): failed to mknod("/etc/fail2ban/action.d", S_IFCHR, 0): no such file or directory
The folder action.d isn't supposed to be there anymore.
Nothing special, here is the gitlab ci/cd, but i get the same error running it manually
make_fail2ban:
stage: make_fail2ban
image:
name: gcr.io/kaniko-project/executor:v1.9.1-debug
entrypoint: [""]
script:
- /kaniko/executor -c /workspace --dockerfile https://mygitlab/mydockerfile --destination "${CI_REGISTRY_IMAGE}/fail2ban:${CI_COMMIT_TAG}"
gcr.io/kaniko-project/executor:v1.9.1-debug
sha256:ac169723b2076f9d5804f4bc05c98397e286da6fdcdd5a09fdc179f06ccb3be1
Triage Notes for the Maintainers
--cacheflagThe text was updated successfully, but these errors were encountered: