Kaniko build fails on apt-get reporting "unable to fetch libraries" #793
Comments
|
Hey @vguaglione, We have an example of dockerfile which does Can you do |
tejal29
added
kind/question
|
Hi @tejal29 I'm not sure in what way one isolated example that happens to work is supposed to help in solving the original problem, but since the issue is tagged as The evidence shows that Here's a simple FROM tensorflow/tensorflow:2.0.0-py3
RUN set -ex; \
date; \
ls -l /var/lib/apt/lists; \
export DEBIAN_FRONTEND=noninteractive; \
apt-get update; \
apt-get install -y tzdataPassing this to # docker run --rm -v $PWD:/ctx gcr.io/kaniko-project/executor -c /ctx --no-push
Unable to find image 'gcr.io/kaniko-project/executor:latest' locally
latest: Pulling from kaniko-project/executor
7ceec777bfab: Pull complete
89b1463f7047: Pull complete
cc4afb9771ac: Pull complete
ea2021d64bdf: Pull complete
8b5a8dfd3fc6: Pull complete
6025730b4535: Pull complete
78f26cbad0a5: Pull complete
Digest: sha256:9c40a04cf1bc9d886f7f000e0b7fa5300c31c89e2ad001e97eeeecdce9f07a29
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest
INFO[0000] Resolved base name tensorflow/tensorflow:2.0.0-py3 to tensorflow/tensorflow:2.0.0-py3
INFO[0000] Resolved base name tensorflow/tensorflow:2.0.0-py3 to tensorflow/tensorflow:2.0.0-py3
INFO[0000] Downloading base image tensorflow/tensorflow:2.0.0-py3
INFO[0001] Error while retrieving image from cache: getting file info: stat /cache/sha256:0b236338fac6c3361cf3ae1448f8c053994e260c1edc4fa63ed80adb3045abb2: no such file or directory
INFO[0001] Downloading base image tensorflow/tensorflow:2.0.0-py3
INFO[0002] Built cross stage deps: map[]
INFO[0002] Downloading base image tensorflow/tensorflow:2.0.0-py3
INFO[0002] Error while retrieving image from cache: getting file info: stat /cache/sha256:0b236338fac6c3361cf3ae1448f8c053994e260c1edc4fa63ed80adb3045abb2: no such file or directory
INFO[0002] Downloading base image tensorflow/tensorflow:2.0.0-py3
INFO[0003] Unpacking rootfs as cmd RUN set -ex; date; ls -l /var/lib/apt/lists; export DEBIAN_FRONTEND=noninteractive; apt-get update; apt-get install -y tzdata requires it.
INFO[0032] Taking snapshot of full filesystem...
INFO[0164] RUN set -ex; date; ls -l /var/lib/apt/lists; export DEBIAN_FRONTEND=noninteractive; apt-get update; apt-get install -y tzdata
INFO[0164] cmd: /bin/sh
INFO[0164] args: [-c set -ex; date; ls -l /var/lib/apt/lists; export DEBIAN_FRONTEND=noninteractive; apt-get update; apt-get install -y tzdata]
+ date
Thu Nov 14 01:28:32 UTC 2019
+ ls -l /var/lib/apt/lists
total 26652
-rw-r--r-- 1 root root 74562 Nov 14 01:26 archive.ubuntu.com_ubuntu_dists_bionic-backports_InRelease
-rw-r--r-- 1 root root 3676 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-backports_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 6359 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-backports_universe_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 88722 Nov 14 01:26 archive.ubuntu.com_ubuntu_dists_bionic-updates_InRelease
-rw-r--r-- 1 root root 1575339 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 13073 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-updates_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 32771 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-updates_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 2120004 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic-updates_universe_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 242048 Nov 14 01:26 archive.ubuntu.com_ubuntu_dists_bionic_InRelease
-rw-r--r-- 1 root root 2165377 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 292075 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 20606 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 18132054 Nov 14 01:25 archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_Packages.lz4
drwxr-xr-x 2 _apt root 4096 Nov 14 01:25 auxfiles
-rw-r----- 1 root root 0 Nov 14 01:26 lock
drwx------ 2 _apt root 4096 Nov 14 01:25 partial
-rw-r--r-- 1 root root 88724 Nov 14 01:26 security.ubuntu.com_ubuntu_dists_bionic-security_InRelease
-rw-r--r-- 1 root root 1093781 Nov 14 01:25 security.ubuntu.com_ubuntu_dists_bionic-security_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 8255 Nov 14 01:25 security.ubuntu.com_ubuntu_dists_bionic-security_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 16374 Nov 14 01:25 security.ubuntu.com_ubuntu_dists_bionic-security_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 1266988 Nov 14 01:25 security.ubuntu.com_ubuntu_dists_bionic-security_universe_binary-amd64_Packages.lz4
+ export DEBIAN_FRONTEND=noninteractive
+ apt-get update
Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease
Hit:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease
Reading package lists...
+ apt-get install -y tzdata
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
tzdata
0 upgraded, 1 newly installed, 0 to remove and 6 not upgraded.
Need to get 190 kB of archives.
After this operation, 3106 kB of additional disk space will be used.
Ign:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 tzdata all 2019b-0ubuntu0.18.04
Err:1 http://security.ubuntu.com/ubuntu bionic-updates/main amd64 tzdata all 2019b-0ubuntu0.18.04
404 Not Found [IP: 91.189.88.31 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/t/tzdata/tzdata_2019b-0ubuntu0.18.04_all.deb 404 Not Found [IP: 91.189.88.31 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
error building image: error building stage: waiting for process to exit: exit status 100For comparison, Docker preserves the original timestamps, which is obviously the sane thing to do and allows # docker run --rm tensorflow/tensorflow:2.0.0-py3 ls -l /var/lib/apt/lists
total 26652
-rw-r--r-- 1 root root 74562 Sep 26 18:55 archive.ubuntu.com_ubuntu_dists_bionic-backports_InRelease
-rw-r--r-- 1 root root 3676 May 29 13:40 archive.ubuntu.com_ubuntu_dists_bionic-backports_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 6359 Sep 14 06:13 archive.ubuntu.com_ubuntu_dists_bionic-backports_universe_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 88722 Oct 2 16:07 archive.ubuntu.com_ubuntu_dists_bionic-updates_InRelease
-rw-r--r-- 1 root root 1575339 Oct 2 13:17 archive.ubuntu.com_ubuntu_dists_bionic-updates_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 13073 Oct 2 13:01 archive.ubuntu.com_ubuntu_dists_bionic-updates_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 32771 Sep 30 22:25 archive.ubuntu.com_ubuntu_dists_bionic-updates_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 2120004 Oct 2 13:17 archive.ubuntu.com_ubuntu_dists_bionic-updates_universe_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 242048 Apr 26 2018 archive.ubuntu.com_ubuntu_dists_bionic_InRelease
-rw-r--r-- 1 root root 2165377 Apr 26 2018 archive.ubuntu.com_ubuntu_dists_bionic_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 292075 Apr 26 2018 archive.ubuntu.com_ubuntu_dists_bionic_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 20606 Apr 24 2018 archive.ubuntu.com_ubuntu_dists_bionic_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 18132054 Apr 26 2018 archive.ubuntu.com_ubuntu_dists_bionic_universe_binary-amd64_Packages.lz4
drwxr-xr-x 1 _apt root 4096 Oct 2 17:35 auxfiles
-rw-r----- 1 root root 0 Oct 2 17:35 lock
drwx------ 1 _apt root 4096 Oct 2 17:36 partial
-rw-r--r-- 1 root root 88724 Oct 2 16:59 security.ubuntu.com_ubuntu_dists_bionic-security_InRelease
-rw-r--r-- 1 root root 1093781 Oct 2 11:39 security.ubuntu.com_ubuntu_dists_bionic-security_main_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 8255 Oct 2 11:39 security.ubuntu.com_ubuntu_dists_bionic-security_multiverse_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 16374 Oct 1 13:02 security.ubuntu.com_ubuntu_dists_bionic-security_restricted_binary-amd64_Packages.lz4
-rw-r--r-- 1 root root 1266988 Oct 2 11:39 security.ubuntu.com_ubuntu_dists_bionic-security_universe_binary-amd64_Packages.lz4 |
|
@andraxin Thanks for providing this example. We have a few container builds failing that similarly reflect what you present above. |
|
I'm having an issue similar to this one and I believe the cause is the same. My build fails when running FROM continuumio/miniconda3
# current latest tag is continuumio/miniconda:4.7.12
RUN apt-get update -y
...This results in the following error: When the same Dockerfile is run locally using |
|
Can this issue be assigned a priority level? I think the current evidence is quite clear. I would say this issue has quite high priority, since basically |
tejal29
added
the
priority/p1
|
@hugokerstens Sorry we missed this in triage. |
|
@tejal29 What's the ETA for release? |
|
got the same issue |
cvgw
changed the title
cvgw
added
in progress
and removed
priority/awaiting_evidence
|
I'm not able to repro this issue using v0.15.0, can someone please supply an example?
|
cvgw
added
needs-reproduction
norepro
I can repo it using v0.15.0. Same behavior observed using v0.16.0 also: Dockerfile: FROM botpress/server:v12_0_0 RUN apt-get update RUN cd /botpress && npm install && npm install node-sp-auth --save-dev && npm install sp-request --save-dev && npm install activedirectory --save-dev EXPOSE 3000-55000 Kaniko build command: docker run -v $(pwd):/tmp gcr.io/kaniko-project/executor:v0.15.0 --dockerfile=/tmp/Dockerfile --context=/tmp --no-push Kaniko Output: INFO[0000] Resolved base name botpress/server:v12_0_0 to botpress/server:v12_0_0 |
|
Thanks @vguaglione I was able to repro with that example. I also tested it with Pr #981 and that seems to fix it |
cvgw
removed
needs-reproduction
norepro
Great news. Will this be merged into the latest and debug versions of the executor? We are currently using both version within our environment. |
Yes, AFAIK we always build both images with all changes |
|
Thanks! |
|
@andraxin that kind of comment is uncalled for. Refer to the code of conduct for more info https://github.com/GoogleContainerTools/kaniko/blob/master/code-of-conduct.md |
Wow... wasn't that the exact same example as in the original thread starter? |
Yes but I needed to test with the same version/tag that @cvgw was using to verify the error. Original results were obtained using the debug version of the executor. |
|
@vguaglione I tried the below dockerfile inside kaniko conatiner with fix #981 The build was successful I just published the following images, Can you try this image on your end? |
|
Both executor:793 and executor:debug_793 return the following error: The warmer:793 does not recognize the --dockerfile option so I was unable to test with this image, using the same command from above. |
|
@vguaglione I re-verified your The dockerfile i used it From the logs looks like your have issues with |
For this test, as with all previous tests, I am running on a mac workstation. I can, however, test this for you in a k8s cluster, if you'd like. Not sure if this is a valid test considering the same Dockerfile works when building the image via 'docker build' on this mac workstation. Please advise. |
|
@vguaglione i am running this using the |
I am using the docker run command on my local workstation, which is a MacBook Pro running Mojave 10.14.6. docker run -v $(pwd):/tmp gcr.io/kaniko-project/executor:latest ... etc |
|
I think this has something to do with mounting the context at
This fails This succeeds @vguaglione can you try mounting your context at Dockerfile used |
|
I can confirm that the image gcr.io/kaniko-project/executor:debug_793 fix the issue in my case. |
|
ok cool! Thanks @olivierboudet ! |
Yes, changing the context to something other than /tmp fixed the problem. Tested with :793 and :debug_793. |
|
Thanks @vguaglione 👍 Are we ok to close this issue now? Opened #999 to track issue with mounting context at |
I believe so. Will this be merged into executor:debug? |
Yes it will |
|
Same dockerfile as used previously to provide an example of failure: FROM botpress/server:v12_0_0 RUN apt-get update RUN cd /botpress && npm install && npm install node-sp-auth --save-dev && npm install sp-request --save-dev && npm install activedirectory --save-dev EXPOSE 3000-55000 @cvgw @tejal29 Executor:debug_793 is successful, executor:debug fails with this output: INFO[0000] Resolved base name botpress/server:v12_0_0 to botpress/server:v12_0_0 |
|
@vguaglione can you share the command you are using to run kaniko? Not able to repro with tag
Dockerfile command |
|
@cvgw Ok, I learned that the 0.15.x version fails as does the debug version. The 0.17.0 and 0.17.1 versions complete successfully. Command is: docker run -v $(pwd):/workspace gcr.io/kaniko-project/executor:debug-v0.17.1 --dockerfile=/workspace/Dockerfile --context=/workspace --no-push Can you tell me when the 0.17.1 version will be merged into debug? For now, we will point to 0.17.1. Vince |
It has already been merged into |
|
@cvgw Ok, I got it. I had to nuke the executor:debug version on my machine and pull a new one and that solved the issue. I think we're good and we can put this issue to rest. |
Actual behavior
Kaniko build fails on apt-get reporting "unable to fetch libraries" while the same build succeed under 'docker build'.
A changing the docker file to rebuild the package cache per this documentation (https://sites.google.com/site/longphanwiki/basic-skills/ubuntu-linux) will solve the problem, however, these directives are not needed when running the build via docker build.
Expected behavior
The container is built successfully.
To Reproduce
Steps to reproduce the behavior:
Additional Information
The dockerfile that's failing under kaniko is defined below:
N/A
gcr.io/kaniko-project/executor:debug
Notes:
If you add the following directives before the apt-get install, kaniko will build the container:
Question:
Is this behavior anticipated? If not, is this a bug in kaniko?
The text was updated successfully, but these errors were encountered: