Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Image pushes with debug, but not with v1.6.0 #1667

Closed
rwong2888 opened this issue Jun 9, 2021 · 5 comments
Closed

Image pushes with debug, but not with v1.6.0 #1667

rwong2888 opened this issue Jun 9, 2021 · 5 comments

Comments

@rwong2888
Copy link

This fails

docker run \
    -v "$HOME"/.config/gcloud:/root/.config/gcloud \
    -v ${PWD}:/workspace \
    gcr.io/kaniko-project/executor:v1.6.0 \
    --dockerfile /workspace/Dockerfile \
    --destination "us.gcr.io/<redacted>/argocd:v2.0.3" \
    --context dir:///workspace/

This works

docker run \
    -v "$HOME"/.config/gcloud:/root/.config/gcloud \
    -v ${PWD}:/workspace \
    gcr.io/kaniko-project/executor:debug \
    --dockerfile /workspace/Dockerfile \
    --destination "us.gcr.io/<redacted>/argocd:v2.0.3" \
    --context dir:///workspace/

Dockerfile

FROM argoproj/argocd:v2.0.3

USER root

RUN apt-get update -y && apt-get install -y \
        curl

USER argocd

Logs with v1.6.0

INFO[0000] Retrieving image manifest argoproj/argocd:v2.0.3
INFO[0000] Retrieving image argoproj/argocd:v2.0.3 from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest argoproj/argocd:v2.0.3
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Unpacking rootfs as cmd RUN apt-get update -y && apt-get install -y         curl requires it.
INFO[0021] USER root
INFO[0021] cmd: USER
INFO[0021] RUN apt-get update -y && apt-get install -y         curl
INFO[0021] Taking snapshot of full filesystem...
INFO[0025] cmd: /bin/sh
INFO[0025] args: [-c apt-get update -y && apt-get install -y         curl]
INFO[0025] util.Lookup returned: &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root}
INFO[0025] performing slow lookup of group ids for root
INFO[0025] Running: [/bin/sh -c apt-get update -y && apt-get install -y         curl]
Get:1 http://security.ubuntu.com/ubuntu groovy-security InRelease [110 kB]
Get:2 http://security.ubuntu.com/ubuntu groovy-security/universe amd64 Packages [119 kB]
Get:3 http://security.ubuntu.com/ubuntu groovy-security/multiverse amd64 Packages [10.9 kB]
Get:4 http://security.ubuntu.com/ubuntu groovy-security/main amd64 Packages [408 kB]
Get:5 http://archive.ubuntu.com/ubuntu groovy InRelease [267 kB]
Get:6 http://security.ubuntu.com/ubuntu groovy-security/restricted amd64 Packages [226 kB]
Get:7 http://archive.ubuntu.com/ubuntu groovy-updates InRelease [115 kB]
Get:8 http://archive.ubuntu.com/ubuntu groovy-backports InRelease [101 kB]
Get:9 http://archive.ubuntu.com/ubuntu groovy/main amd64 Packages [1768 kB]
Get:10 http://archive.ubuntu.com/ubuntu groovy/restricted amd64 Packages [87.5 kB]
Get:11 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages [16.1 MB]
Get:12 http://archive.ubuntu.com/ubuntu groovy/multiverse amd64 Packages [247 kB]
Get:13 http://archive.ubuntu.com/ubuntu groovy-updates/universe amd64 Packages [245 kB]
Get:14 http://archive.ubuntu.com/ubuntu groovy-updates/restricted amd64 Packages [256 kB]
Get:15 http://archive.ubuntu.com/ubuntu groovy-updates/multiverse amd64 Packages [17.8 kB]
Get:16 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages [616 kB]
Get:17 http://archive.ubuntu.com/ubuntu groovy-backports/universe amd64 Packages [4223 B]
Fetched 20.7 MB in 30s (681 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libcurl4
The following NEW packages will be installed:
  curl libcurl4
0 upgraded, 2 newly installed, 0 to remove and 8 not upgraded.
Need to get 396 kB of archives.
After this operation, 1119 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu4.3 [235 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 curl amd64 7.68.0-1ubuntu4.3 [161 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 396 kB in 1s (462 kB/s)
Selecting previously unselected package libcurl4:amd64.
(Reading database ... 16093 files and directories currently installed.)
Preparing to unpack .../libcurl4_7.68.0-1ubuntu4.3_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu4.3) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.68.0-1ubuntu4.3_amd64.deb ...
Unpacking curl (7.68.0-1ubuntu4.3) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu4.3) ...
Setting up curl (7.68.0-1ubuntu4.3) ...
Processing triggers for libc-bin (2.32-0ubuntu3) ...
INFO[0059] Taking snapshot of full filesystem...
INFO[0062] USER argocd
INFO[0062] cmd: USER
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7fb07e1300c4]

runtime stack:
runtime.throw(0x7e0d4b, 0x2a)
	/usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.sigpanic()
	/usr/local/go/src/runtime/signal_unix.go:726 +0x4ac

goroutine 1 [syscall]:
runtime.cgocall(0x6a8ae0, 0xc000042a30, 0xc0000b6028)
	/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc000042a00 sp=0xc0000429c8 pc=0x40563b
os/user._Cfunc_mygetpwuid_r(0x0, 0xc00009ced0, 0x18dc070, 0x400, 0xc0000b6028, 0x7fb000000000)
	_cgo_gotypes.go:175 +0x4d fp=0xc000042a30 sp=0xc000042a00 pc=0x68b1ad
os/user.lookupUnixUid.func1.1(0x0, 0xc00009ced0, 0xc000098dc0, 0xc0000b6028, 0xc000042ad0)
	/usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0xd0 fp=0xc000042a80 sp=0xc000042a30 pc=0x68bf50
os/user.lookupUnixUid.func1(0x798840)
	/usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0x45 fp=0xc000042ab8 sp=0xc000042a80 pc=0x68bfc5
os/user.retryWithBuffer(0xc000098dc0, 0xc000042b90, 0x41a165, 0x7fb0a76942f8)
	/usr/local/go/src/os/user/cgo_lookup_unix.go:247 +0x3e fp=0xc000042b10 sp=0xc000042ab8 pc=0x68bcde
os/user.lookupUnixUid(0x0, 0x0, 0x0, 0x0)
	/usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc000042bd8 sp=0xc000042b10 pc=0x68b5f2
os/user.current(0xc000042c58, 0x4d419c, 0xc00009f220)
	/usr/local/go/src/os/user/cgo_lookup_unix.go:49 +0x49 fp=0xc000042c18 sp=0xc000042bd8 pc=0x68b489
os/user.Current.func1()
	/usr/local/go/src/os/user/lookup.go:15 +0x25 fp=0xc000042c40 sp=0xc000042c18 pc=0x68be05
sync.(*Once).doSlow(0xa09c40, 0x7ed020)
	/usr/local/go/src/sync/once.go:66 +0xec fp=0xc000042c90 sp=0xc000042c40 pc=0x474cac
sync.(*Once).Do(...)
	/usr/local/go/src/sync/once.go:57
os/user.Current(0x47707b, 0xa41360, 0xc00009c5d0)
	/usr/local/go/src/os/user/lookup.go:15 +0x105 fp=0xc000042cc0 sp=0xc000042c90 pc=0x68b005
github.com/GoogleCloudPlatform/docker-credential-gcr/util.unixHomeDir(0x4d4800, 0xc00009f220)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/util/util.go:42 +0x25 fp=0xc000042cf0 sp=0xc000042cc0 pc=0x68e7a5
github.com/GoogleCloudPlatform/docker-credential-gcr/util.SdkConfigPath(0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/util/util.go:34 +0x26 fp=0xc000042d58 sp=0xc000042cf0 pc=0x68e686
github.com/GoogleCloudPlatform/docker-credential-gcr/store.dockerCredentialPath(0x7fb0a76887d0, 0xc000098db0, 0x7fb07f1f4228, 0x18)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/store/store.go:215 +0x6d fp=0xc000042dc8 sp=0xc000042d58 pc=0x691a8d
github.com/GoogleCloudPlatform/docker-credential-gcr/store.DefaultGCRCredStore(...)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/store/store.go:84
github.com/GoogleCloudPlatform/docker-credential-gcr/cli.(*helperCmd).Execute(0xc0000ae300, 0x8393e0, 0xc0000a2000, 0xc0000b8360, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/cli/dockerHelper.go:35 +0x35 fp=0xc000042e78 sp=0xc000042dc8 pc=0x6a6b35
github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands.(*Commander).Execute(0xc0000f2000, 0x8393e0, 0xc0000a2000, 0x0, 0x0, 0x0, 0x39)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands/subcommands.go:209 +0x30d fp=0xc000042f20 sp=0xc000042e78 pc=0x69274d
github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands.Execute(...)
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/vendor/github.com/google/subcommands/subcommands.go:492
main.main()
	/go/src/github.com/GoogleCloudPlatform/docker-credential-gcr/main.go:54 +0x63f fp=0xc000042f88 sp=0xc000042f20 pc=0x6a87df
runtime.main()
	/usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc000042fe0 sp=0xc000042f88 pc=0x439c89
runtime.goexit()
	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000042fe8 sp=0xc000042fe0 pc=0x46b861
error pushing image: resolving pushAuth: error getting credentials - err: exit status 2, out: ``

Logs with debug

INFO[0000] Retrieving image manifest argoproj/argocd:v2.0.3
INFO[0000] Retrieving image argoproj/argocd:v2.0.3 from registry index.docker.io
INFO[0001] Built cross stage deps: map[]
INFO[0001] Retrieving image manifest argoproj/argocd:v2.0.3
INFO[0001] Returning cached image manifest
INFO[0001] Executing 0 build triggers
INFO[0001] Unpacking rootfs as cmd RUN apt-get update -y && apt-get install -y         curl requires it.
INFO[0021] USER root
INFO[0021] cmd: USER
INFO[0021] RUN apt-get update -y && apt-get install -y         curl
INFO[0021] Taking snapshot of full filesystem...
INFO[0025] cmd: /bin/sh
INFO[0025] args: [-c apt-get update -y && apt-get install -y         curl]
INFO[0025] util.Lookup returned: &{Uid:0 Gid:0 Username:root Name:root HomeDir:/root}
INFO[0025] performing slow lookup of group ids for root
INFO[0025] Running: [/bin/sh -c apt-get update -y && apt-get install -y         curl]
Get:1 http://security.ubuntu.com/ubuntu groovy-security InRelease [110 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy InRelease [267 kB]
Get:3 http://security.ubuntu.com/ubuntu groovy-security/restricted amd64 Packages [226 kB]
Get:4 http://security.ubuntu.com/ubuntu groovy-security/universe amd64 Packages [119 kB]
Get:5 http://security.ubuntu.com/ubuntu groovy-security/multiverse amd64 Packages [10.9 kB]
Get:6 http://security.ubuntu.com/ubuntu groovy-security/main amd64 Packages [408 kB]
Get:7 http://archive.ubuntu.com/ubuntu groovy-updates InRelease [115 kB]
Get:8 http://archive.ubuntu.com/ubuntu groovy-backports InRelease [101 kB]
Get:9 http://archive.ubuntu.com/ubuntu groovy/universe amd64 Packages [16.1 MB]
Get:10 http://archive.ubuntu.com/ubuntu groovy/multiverse amd64 Packages [247 kB]
Get:11 http://archive.ubuntu.com/ubuntu groovy/main amd64 Packages [1768 kB]
Get:12 http://archive.ubuntu.com/ubuntu groovy/restricted amd64 Packages [87.5 kB]
Get:13 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 Packages [616 kB]
Get:14 http://archive.ubuntu.com/ubuntu groovy-updates/universe amd64 Packages [245 kB]
Get:15 http://archive.ubuntu.com/ubuntu groovy-updates/restricted amd64 Packages [256 kB]
Get:16 http://archive.ubuntu.com/ubuntu groovy-updates/multiverse amd64 Packages [17.8 kB]
Get:17 http://archive.ubuntu.com/ubuntu groovy-backports/universe amd64 Packages [4223 B]
Fetched 20.7 MB in 32s (654 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  libcurl4
The following NEW packages will be installed:
  curl libcurl4
0 upgraded, 2 newly installed, 0 to remove and 8 not upgraded.
Need to get 396 kB of archives.
After this operation, 1119 kB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 libcurl4 amd64 7.68.0-1ubuntu4.3 [235 kB]
Get:2 http://archive.ubuntu.com/ubuntu groovy-updates/main amd64 curl amd64 7.68.0-1ubuntu4.3 [161 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 396 kB in 1s (330 kB/s)
Selecting previously unselected package libcurl4:amd64.
(Reading database ... 16093 files and directories currently installed.)
Preparing to unpack .../libcurl4_7.68.0-1ubuntu4.3_amd64.deb ...
Unpacking libcurl4:amd64 (7.68.0-1ubuntu4.3) ...
Selecting previously unselected package curl.
Preparing to unpack .../curl_7.68.0-1ubuntu4.3_amd64.deb ...
Unpacking curl (7.68.0-1ubuntu4.3) ...
Setting up libcurl4:amd64 (7.68.0-1ubuntu4.3) ...
Setting up curl (7.68.0-1ubuntu4.3) ...
Processing triggers for libc-bin (2.32-0ubuntu3) ...
INFO[0060] Taking snapshot of full filesystem...
INFO[0062] USER argocd
INFO[0062] cmd: USER
INFO[0062] Pushing image to us.gcr.io/forbes-development/argocd:v2.0.3
INFO[0071] Pushed image to 1 destinations
@campbellmc
Copy link

Seeing same thing using trigger to build on gcloud using latest. Can send build log if required.

@lewiscowleschipuk
Copy link

We are also seeing images not push, but with no notification in gitlab builds using this tool. Unfortunately we do not seem to be receiving any error message.

@elatt
Copy link

elatt commented Jul 1, 2021 •
edited

I'm seeing this issue with gcr.io/kaniko-project/executor:v1.6.0 and gcr.io/kaniko-project/executor:v1.5.2 but as reported, everything works fine with gcr.io/kaniko-project/executor:v1.6.0-debug

Not sure why there is such a difference between the GCR helper in the debug image vs the main image:

@seankhliao
Copy link

Also running into this pushing an image based on archlinux:base

The credential helper in the main image is built with CGo enabled and statically linked while the debug image downloads it from remote, which is built without Cgo

This is a problem when certain C library functions aren't available, demonstrated by the example (run with docker):

package main

import (
	"fmt"
	"os/user"
)

func main() {
	fmt.Println(user.Current())
}

and a dockerfile:

FROM golang:1.15
WORKDIR /workspace
COPY . .
RUN go build -ldflags "-linkmode external -extldflags -static" -i -o usertest .
RUN ./usertest

FROM archlinux:base
WORKDIR /workspace
COPY --from=0 /workspace/usertest /workspace/usertest
RUN ./usertest

This gives a warning:

#11 [stage-0 4/5] RUN go build -ldflags "-linkmode external -extldflags -static" -i -o usertest .
#11 sha256:c7cf070e1983317dadae2fc99c19e4710d8626b61e760f995fa3ef0b915db7eb
#11 1.864 # example
#11 1.864 /usr/bin/ld: /tmp/go-link-007306595/000002.o: in function `mygetgrouplist':
#11 1.864 /usr/local/go/src/os/user/getgrouplist_unix.go:16: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
#11 1.864 /usr/bin/ld: /tmp/go-link-007306595/000001.o: in function `mygetgrgid_r':
#11 1.864 /usr/local/go/src/os/user/cgo_lookup_unix.go:38: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
#11 1.864 /usr/bin/ld: /tmp/go-link-007306595/000001.o: in function `mygetgrnam_r':
#11 1.864 /usr/local/go/src/os/user/cgo_lookup_unix.go:43: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
#11 1.864 /usr/bin/ld: /tmp/go-link-007306595/000001.o: in function `mygetpwnam_r':
#11 1.864 /usr/local/go/src/os/user/cgo_lookup_unix.go:33: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
#11 1.864 /usr/bin/ld: /tmp/go-link-007306595/000001.o: in function `mygetpwuid_r':
#11 1.864 /usr/local/go/src/os/user/cgo_lookup_unix.go:28: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
#11 DONE 2.0s

and a panic

#14 [stage-1 4/4] RUN ./usertest
#14 sha256:697062f954cd6ff206ddacf2d826503f06ed30a8e94e50b01403f680664c4d51
#14 0.876 fatal error: unexpected signal during runtime execution
#14 0.879 [signal SIGSEGV: segmentation violation code=0x1 addr=0xe5 pc=0x7f311e101414]
#14 0.879
#14 0.879 runtime stack:
#14 0.879 runtime.throw(0x56417e, 0x2a)
#14 0.879 	/usr/local/go/src/runtime/panic.go:1116 +0x72
#14 0.879 runtime.sigpanic()
#14 0.879 	/usr/local/go/src/runtime/signal_unix.go:726 +0x4ac
#14 0.879
#14 0.879 goroutine 1 [syscall]:
#14 0.879 runtime.cgocall(0x49f6d0, 0xc000036c88, 0xc0000b6018)
#14 0.879 	/usr/local/go/src/runtime/cgocall.go:133 +0x5b fp=0xc000036c58 sp=0xc000036c20 pc=0x404fdb
#14 0.879 os/user._Cfunc_mygetpwuid_r(0x0, 0xc00009a150, 0x1a1d070, 0x400, 0xc0000b6018, 0x7f3100000000)
#14 0.879 	_cgo_gotypes.go:175 +0x4d fp=0xc000036c88 sp=0xc000036c58 pc=0x49e26d
#14 0.879 os/user.lookupUnixUid.func1.1(0x0, 0xc00009a150, 0xc0000981e0, 0xc0000b6018, 0xc000036d28)
#14 0.879 	/usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0xd0 fp=0xc000036cd8 sp=0xc000036c88 pc=0x49f010
#14 0.879 os/user.lookupUnixUid.func1(0x54c860)
#14 0.879 	/usr/local/go/src/os/user/cgo_lookup_unix.go:103 +0x45 fp=0xc000036d10 sp=0xc000036cd8 pc=0x49f085
#14 0.879 os/user.retryWithBuffer(0xc0000981e0, 0xc000036de8, 0x544ce0, 0x3)
#14 0.879 	/usr/local/go/src/os/user/cgo_lookup_unix.go:247 +0x3e fp=0xc000036d68 sp=0xc000036d10 pc=0x49ed9e
#14 0.879 os/user.lookupUnixUid(0x0, 0x0, 0x0, 0x0)
#14 0.879 	/usr/local/go/src/os/user/cgo_lookup_unix.go:96 +0x132 fp=0xc000036e30 sp=0xc000036d68 pc=0x49e6b2
#14 0.879 os/user.current(0x0, 0x1, 0xc000012000)
#14 0.879 	/usr/local/go/src/os/user/cgo_lookup_unix.go:49 +0x49 fp=0xc000036e70 sp=0xc000036e30 pc=0x49e549
#14 0.879 os/user.Current.func1()
#14 0.879 	/usr/local/go/src/os/user/lookup.go:15 +0x25 fp=0xc000036e98 sp=0xc000036e70 pc=0x49eec5
#14 0.879 sync.(*Once).doSlow(0x6274e0, 0x565728)
#14 0.879 	/usr/local/go/src/sync/once.go:66 +0xec fp=0xc000036ee8 sp=0xc000036e98 pc=0x47450c
#14 0.880 sync.(*Once).Do(...)
#14 0.880 	/usr/local/go/src/sync/once.go:57
#14 0.880 os/user.Current(0xc000036f48, 0x443df7, 0x61a0a0)
#14 0.880 	/usr/local/go/src/os/user/lookup.go:15 +0x105 fp=0xc000036f18 sp=0xc000036ee8 pc=0x49e0c5
#14 0.880 main.main()
#14 0.880 	/workspace/main.go:9 +0x26 fp=0xc000036f88 sp=0xc000036f18 pc=0x49f526
#14 0.880 runtime.main()
#14 0.880 	/usr/local/go/src/runtime/proc.go:204 +0x209 fp=0xc000036fe0 sp=0xc000036f88 pc=0x436ee9
#14 0.880 runtime.goexit()
#14 0.880 	/usr/local/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc000036fe8 sp=0xc000036fe0 pc=0x4658a1
#14 ERROR: executor failed running [/bin/sh -c ./usertest]: exit code: 2

@bygui86 bygui86 mentioned this issue Dec 15, 2021
Open
@rwong2888
Copy link
Author

Using updated image works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@elatt @campbellmc @seankhliao @rwong2888 @lewiscowleschipuk