BUG: podman-remote-static crash in ubuntu 20.04 (wsl) #13557

reini-1 · 2022-03-18T15:52:53Z

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

WSL~Ubuntu-20.04> reini@wsl:~/Downloads$ tar -xzvf podman-remote-static.tar.gz
podman-remote-static

WSL~Ubuntu-20.04> reini@wsl:~/Downloads$ ./podman-remote-static --version
podman-remote-static: dl-call-libc-early-init.c:37: _dl_call_libc_early_init: Assertion `sym != NULL' failed.
SIGABRT: abort
PC=0xf2568b m=0 sigcode=18446744073709551610
signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0xf07460, 0xc00069f4d8)
        /usr/lib/golang/src/runtime/cgocall.go:157 +0x5c fp=0xc00069f4b0 sp=0xc00069f478 pc=0x4059dc
os/user._Cfunc_mygetpwnam_r(0xc0005ab940, 0xc00059ed50, 0x396fd00, 0x400, 0xc0005ac288)
        _cgo_gotypes.go:158 +0x4c fp=0xc00069f4d8 sp=0xc00069f4b0 pc=0x72bb6c
os/user.lookupUser.func1.1({0xc0005ab940, 0xc0004bf4f0?, 0x0?}, 0x1c06480?, 0xc00069f580?, 0x40f927?)
        /usr/lib/golang/src/os/user/cgo_lookup_unix.go:69 +0xbb fp=0xc00069f538 sp=0xc00069f4d8 pc=0x72c45b
os/user.lookupUser.func1()
        /usr/lib/golang/src/os/user/cgo_lookup_unix.go:69 +0x31 fp=0xc00069f578 sp=0xc00069f538 pc=0x72c371
os/user.retryWithBuffer(0xc0004bf4f0, 0xc00069f668)
        /usr/lib/golang/src/os/user/cgo_lookup_unix.go:244 +0x39 fp=0xc00069f5c0 sp=0xc00069f578 pc=0x72d559
os/user.lookupUser({0x1246b60, 0x7})
        /usr/lib/golang/src/os/user/cgo_lookup_unix.go:60 +0x165 fp=0xc00069f6b8 sp=0xc00069f5c0 pc=0x72c145
os/user.Lookup({0x1246b60, 0x7})
        /usr/lib/golang/src/os/user/lookup.go:36 +0x91 fp=0xc00069f6f8 sp=0xc00069f6b8 pc=0x72b711
github.com/containers/storage/pkg/chrootarchive.init.0()
        /home/lsm5/go/src/github.com/containers/podman/vendor/github.com/containers/storage/pkg/chrootarchive/archive.go:23 +0x25 fp=0xc00069f730 sp=0xc00069f6f8 pc=0xa10885
runtime.doInit(0x1bab2e0)
        /usr/lib/golang/src/runtime/proc.go:6222 +0x126 fp=0xc00069f860 sp=0xc00069f730 pc=0x446aa6
runtime.doInit(0x1bac4e0)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069f990 sp=0xc00069f860 pc=0x4469f1
runtime.doInit(0x1bb08a0)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069fac0 sp=0xc00069f990 pc=0x4469f1
runtime.doInit(0x1ba9060)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069fbf0 sp=0xc00069fac0 pc=0x4469f1
runtime.doInit(0x1baa960)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069fd20 sp=0xc00069fbf0 pc=0x4469f1
runtime.doInit(0x1bad200)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069fe50 sp=0xc00069fd20 pc=0x4469f1
runtime.doInit(0x1bb1820)
        /usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00069ff80 sp=0xc00069fe50 pc=0x4469f1
runtime.main()
        /usr/lib/golang/src/runtime/proc.go:233 +0x1d3 fp=0xc00069ffe0 sp=0xc00069ff80 pc=0x439a53
runtime.goexit()
        /usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00069ffe8 sp=0xc00069ffe0 pc=0x4682a1

rax    0x0
rbx    0xafe
rcx    0xf2568b
rdx    0x6
rdi    0xafe
rsi    0xafe
rbp    0x6
rsp    0x7fff24bf3a70
r8     0x39703d0
r9     0x0
r10    0x8
r11    0x246
r12    0x146a94a
r13    0x25
r14    0x145a7b3
r15    0x3972370
rip    0xf2568b
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

old version works fine

WSL~Ubuntu-20.04> reini@wsl:~/Downloads$ tar -xzvf podman-remote-static-3.4.4.tar.gz
podman-remote-static

WSL~Ubuntu-20.04> reini@wsl:~/Downloads$ ./podman-remote-static --version
podman-remote-static version 3.4.4

Steps to reproduce the issue:

download https://github.com/containers/podman/releases/download/v4.0.2/podman-remote-static.tar.gz
un-tar it
run it

Describe the results you received:

crash, see above

Describe the results you expected:

don't crash, show version

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

not possible, see above but should be 4.0.2

Output of podman info --debug:

not possible, see above

Package info (e.g. output of rpm -q podman or apt list podman):

direct download from github, see above

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):

Ubuntu in Windows 11 WSL but did not work in Windows 10 also.

WSL~~Ubuntu-20.04> reini@wsl:~~/Downloads$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.4 LTS
Release: 20.04
Codename: focal

The text was updated successfully, but these errors were encountered:

Luap99 · 2022-03-21T11:13:02Z

@mheon @lsm5 PTAL

mheon · 2022-03-21T13:18:27Z

Initial thought: is the static build actually static? Anyone run ldd against it recently? That looks like a potential libc version mismatch issue.

lsm5 · 2022-03-21T13:43:48Z

@mheon

$ ldd podman-remote-static
	not a dynamic executable

If it matters, I built it on rawhide with go 1.18 (i think it was an rc back then).

afbjorklund · 2022-03-21T13:45:11Z

It is a known bug in Go, when linking statically with glibc

Mentioned here: http://tbg.github.io/golang-static-linking-bug

lsm5 · 2022-03-21T13:45:19Z

using the executable on a centos 9 stream, I see:

$ ./podman-remote-static --version
fatal error: unexpected signal during runtime execution
[signal SIGFPE: floating-point exception code=0x1 addr=0x7f3fb83c9e29 pc=0x7f3fb83c9e29]

runtime stack:
runtime.throw({0x127e3e1?, 0xfba6ea?})
	/usr/lib/golang/src/runtime/panic.go:992 +0x71
runtime.sigpanic()
	/usr/lib/golang/src/runtime/signal_unix.go:802 +0x3a9

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0xf074a0, 0xc00059f428)
	/usr/lib/golang/src/runtime/cgocall.go:157 +0x5c fp=0xc00059f400 sp=0xc00059f3c8 pc=0x4059dc
os/user._Cfunc_mygetpwuid_r(0x3e9, 0xc0004b4f30, 0x2111d00, 0x400, 0xc0000102e0)
	_cgo_gotypes.go:175 +0x4c fp=0xc00059f428 sp=0xc00059f400 pc=0x72bc8c
os/user.lookupUnixUid.func1.1(0x3e9, 0x1c06480?, 0xc00059f4c0?, 0x40f927?)
	/usr/lib/golang/src/os/user/cgo_lookup_unix.go:102 +0xa6 fp=0xc00059f488 sp=0xc00059f428 pc=0x72c966
os/user.lookupUnixUid.func1()
	/usr/lib/golang/src/os/user/cgo_lookup_unix.go:102 +0x29 fp=0xc00059f4b8 sp=0xc00059f488 pc=0x72c8a9
os/user.retryWithBuffer(0xc000047370, 0xc00059f598)
	/usr/lib/golang/src/os/user/cgo_lookup_unix.go:244 +0x39 fp=0xc00059f500 sp=0xc00059f4b8 pc=0x72d559
os/user.lookupUnixUid(0x3e9)
	/usr/lib/golang/src/os/user/cgo_lookup_unix.go:95 +0x10f fp=0xc00059f5d8 sp=0xc00059f500 pc=0x72c6af
os/user.current()
	/usr/lib/golang/src/os/user/cgo_lookup_unix.go:48 +0x47 fp=0xc00059f618 sp=0xc00059f5d8 pc=0x72bfc7
os/user.Current.func1()
	/usr/lib/golang/src/os/user/lookup.go:15 +0x17 fp=0xc00059f628 sp=0xc00059f618 pc=0x72b537
sync.(*Once).doSlow(0x1c06480?, 0x1c?)
	/usr/lib/golang/src/sync/once.go:68 +0xc2 fp=0xc00059f688 sp=0xc00059f628 pc=0x4726c2
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:59
os/user.Current()
	/usr/lib/golang/src/os/user/lookup.go:15 +0x37 fp=0xc00059f6b8 sp=0xc00059f688 pc=0x72b5d7
os/user.Lookup({0x1246b60, 0x7})
	/usr/lib/golang/src/os/user/lookup.go:33 +0x27 fp=0xc00059f6f8 sp=0xc00059f6b8 pc=0x72b6a7
github.com/containers/storage/pkg/chrootarchive.init.0()
	/home/lsm5/go/src/github.com/containers/podman/vendor/github.com/containers/storage/pkg/chrootarchive/archive.go:23 +0x25 fp=0xc00059f730 sp=0xc00059f6f8 pc=0xa10885
runtime.doInit(0x1bab2e0)
	/usr/lib/golang/src/runtime/proc.go:6222 +0x126 fp=0xc00059f860 sp=0xc00059f730 pc=0x446aa6
runtime.doInit(0x1bac4e0)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059f990 sp=0xc00059f860 pc=0x4469f1
runtime.doInit(0x1bb08a0)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059fac0 sp=0xc00059f990 pc=0x4469f1
runtime.doInit(0x1ba9060)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059fbf0 sp=0xc00059fac0 pc=0x4469f1
runtime.doInit(0x1baa960)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059fd20 sp=0xc00059fbf0 pc=0x4469f1
runtime.doInit(0x1bad200)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059fe50 sp=0xc00059fd20 pc=0x4469f1
runtime.doInit(0x1bb1820)
	/usr/lib/golang/src/runtime/proc.go:6199 +0x71 fp=0xc00059ff80 sp=0xc00059fe50 pc=0x4469f1
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:233 +0x1d3 fp=0xc00059ffe0 sp=0xc00059ff80 pc=0x439a53
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00059ffe8 sp=0xc00059ffe0 pc=0x4682a1

mheon · 2022-03-21T13:46:50Z

Ah, lovely. I love Go.

lsm5 · 2022-03-21T13:51:05Z

@afbjorklund @mheon ummm, so do we need to build for each distro-release combination?

afbjorklund · 2022-03-21T17:20:59Z

I'm getting a ton of warnings, when building podman-remote-static.

# github.com/containers/podman/v4/cmd/podman
/usr/bin/ld: /tmp/go-link-1312392923/000031.o: in function `New':
/home/anders/podman/vendor/github.com/miekg/pkcs11/pkcs11.go:75: warning: Using 'dlopen' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000010.o: in function `mygetgrouplist':
/usr/local/go/src/os/user/getgrouplist_unix.go:18: warning: Using 'getgrouplist' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000009.o: in function `mygetgrgid_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:40: warning: Using 'getgrgid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000009.o: in function `mygetgrnam_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:45: warning: Using 'getgrnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000009.o: in function `mygetpwnam_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:35: warning: Using 'getpwnam_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000009.o: in function `mygetpwuid_r':
/usr/local/go/src/os/user/cgo_lookup_unix.go:30: warning: Using 'getpwuid_r' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking
/usr/bin/ld: /tmp/go-link-1312392923/000007.o: in function `_cgo_3c1cec0c9a4e_C2func_getaddrinfo':
/tmp/go-build/cgo-gcc-prolog:58: warning: Using 'getaddrinfo' in statically linked applications requires at runtime the shared libraries from the glibc version used for linking

Probably related to the change of CGO_ENABLED, since v3.4.2 ?

Previously, it was set to CGO_ENABLED=0 for the static build...

mheon · 2022-03-21T17:23:44Z

Why did we do that? I don't know that podman-remote actually needs cgo.

mheon · 2022-03-21T17:24:53Z

ab4c9a8 suggests there is a legitimate reason to want to do it, which is unfortunate

afbjorklund · 2022-03-21T17:25:19Z

Why did we do that?

It was done here: 319fcf5

 $(SRCBINDIR)/podman-remote-static: $(SRCBINDIR) .gopathok $(SOURCES) go.mod go.sum
-       CGO_ENABLED=0 \
-               GOOS=$(GOOS) \
-               $(GO) build \
+       $(GOCMD) build \
                $(BUILDFLAGS) \

afbjorklund · 2022-03-21T17:28:50Z

do we need to build for each distro-release combination?

@lsm5 probably just delete podman-remote-static.tar.gz, similar to the other static build ? (the Nix one)

Previously podman-remote was available in the podman deb.

    podman | 100:3.4.2-1 | https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/xUbuntu_20.04  Packages

lsm5 · 2022-03-21T18:13:30Z

@afbjorklund yes, I find building with musl a PITA too. I'm cool with removing podman-remote-static. @mheon @ashley-cui @baude @rhatdan wdyt?

Luap99 · 2022-03-21T18:16:40Z

Why not just disable cgo, we clearly have users using it.

lsm5 · 2022-03-21T18:39:42Z

@Luap99 do you mean something like:

diff --git a/Makefile b/Makefile
index 523bb30fb..ad0b98651 100644
--- a/Makefile
+++ b/Makefile
@@ -336,6 +336,9 @@ $(SRCBINDIR)/podman$(BINSFX): $(SRCBINDIR) .gopathok $(SOURCES) go.mod go.sum
                -o $@ ./cmd/podman

 $(SRCBINDIR)/podman-remote-static: $(SRCBINDIR) .gopathok $(SOURCES) go.mod go.sum
+       CGO_ENABLED=0 \
+       CGO_CFLAGS="-I/usr/x86_64-linux-musl/include -I/usr/include/linux" \
+       CC=/usr/bin/musl-gcc \
        $(GOCMD) build \
                $(BUILDFLAGS) \
                $(GO_LDFLAGS) '$(LDFLAGS_PODMAN_STATIC)' \

Don't know if that's supposed to work or if musl has a separate set of variables. Without the CGO_CFLAGS, it isn't able to find linux/fs.h and linux/limits.h.

That gives me:

$ make podman-remote-static
CGO_ENABLED=0 \
CGO_CFLAGS="-I/usr/x86_64-linux-musl/include -I/usr/include/linux" \
CC=/usr/bin/musl-gcc \
CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \
	-mod=vendor  \
	-ldflags '-X github.com/containers/podman/v4/libpod/define.gitCommit=e034db16bf68c06c1ecf63addd033fcbd83090e3-dirty -X github.com/containers/podman/v4/libpod/define.buildInfo=1647887797 -X github.com/containers/podman/v4/libpod/config._installPrefix=/usr/local -X github.com/containers/podman/v4/libpod/config._etcDir=/usr/local/etc -X github.com/containers/common/pkg/config.additionalHelperBinariesDir=  -extldflags=-static' \
	-tags "remote exclude_graphdriver_btrfs btrfs_noversion exclude_graphdriver_devicemapper containers_image_openpgp" \
	-o bin/podman-remote-static ./cmd/podman
# runtime/cgo
In file included from _cgo_export.c:4:
cgo-builtin-export-prolog:8:33: error: unknown type name 'ptrdiff_t'
make: *** [Makefile:339: bin/podman-remote-static] Error 2

Luap99 · 2022-03-21T18:45:35Z

Because you still have cgo enabled:

CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \

lsm5 · 2022-03-21T18:50:18Z

Because you still have cgo enabled:

CGO_ENABLED=1 GOOS=linux GOARCH=amd64 go build \

ah I see, that did give me a successful build. Let me test the binary on another env. Thanks!

Resolves: containers#13557 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>

openshift-ci bot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 18, 2022

This comment was marked as outdated.

Sign in to view

lsm5 mentioned this issue Mar 21, 2022

Makefile: build podman-remote-static with cgo disabled #13584

Merged

lsm5 added a commit to lsm5/podman that referenced this issue Mar 21, 2022

Makefile: build podman-remote-static with cgo disabled

8961dd3

Resolves: containers#13557 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>

edsantiago changed the title ~~BUG: portman-remote-static crash in ubuntu 20.04 (wsl)~~ BUG: podman-remote-static crash in ubuntu 20.04 (wsl) Mar 21, 2022

openshift-merge-robot closed this as completed in #13584 Mar 22, 2022

mheon pushed a commit to mheon/libpod that referenced this issue Mar 30, 2022

Makefile: build podman-remote-static with cgo disabled

116a9ef

Resolves: containers#13557 [NO NEW TESTS NEEDED] Signed-off-by: Lokesh Mandvekar <lsm5@fedoraproject.org>

randombenj mentioned this issue May 12, 2022

v4.1.0 static binary is windows not linux #14201

Closed

zhangguanzhang mentioned this issue May 25, 2022

[Linux] Podman static binary for 4.0.2 release have fatal error #14353

Closed

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 20, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 20, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

BUG: podman-remote-static crash in ubuntu 20.04 (wsl) #13557

BUG: podman-remote-static crash in ubuntu 20.04 (wsl) #13557

reini-1 commented Mar 18, 2022

Luap99 commented Mar 21, 2022

mheon commented Mar 21, 2022

lsm5 commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 •

edited

Loading

lsm5 commented Mar 21, 2022

mheon commented Mar 21, 2022

lsm5 commented Mar 21, 2022

This comment was marked as outdated.

afbjorklund commented Mar 21, 2022 •

edited

Loading

mheon commented Mar 21, 2022

mheon commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

lsm5 commented Mar 21, 2022 •

edited

Loading

Luap99 commented Mar 21, 2022

lsm5 commented Mar 21, 2022

Luap99 commented Mar 21, 2022

lsm5 commented Mar 21, 2022

BUG: podman-remote-static crash in ubuntu 20.04 (wsl) #13557

BUG: podman-remote-static crash in ubuntu 20.04 (wsl) #13557

Comments

reini-1 commented Mar 18, 2022

Luap99 commented Mar 21, 2022

mheon commented Mar 21, 2022

lsm5 commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 • edited Loading

lsm5 commented Mar 21, 2022

mheon commented Mar 21, 2022

lsm5 commented Mar 21, 2022

This comment was marked as outdated.

afbjorklund commented Mar 21, 2022 • edited Loading

mheon commented Mar 21, 2022

mheon commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 • edited Loading

afbjorklund commented Mar 21, 2022 • edited Loading

lsm5 commented Mar 21, 2022 • edited Loading

Luap99 commented Mar 21, 2022

lsm5 commented Mar 21, 2022

Luap99 commented Mar 21, 2022

lsm5 commented Mar 21, 2022

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

afbjorklund commented Mar 21, 2022 •

edited

Loading

lsm5 commented Mar 21, 2022 •

edited

Loading