cmd/go: cgo-enabled binary built with Ubuntu 22.04 is incompatible with Ubuntu 20.04

[zeisss]

What version of Go are you using (go version)?

$ ~/go/bin/go version
go version go1.19.4 linux/amd64

Does this issue reproduce with the latest release?

This is the latest stable release from the website.

What operating system and processor architecture are you using (go env)?

go env Output

$ ~/go/bin/go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ubuntu/.cache/go-build"
GOENV="/home/ubuntu/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ubuntu/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ubuntu/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ubuntu/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ubuntu/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.4"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build646395045=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Building a minimal binary (using net/http code) with -trimpath on Ubuntu 22.04 makes it incompatible with Ubuntu 20.04:

ubuntu@jammy:~$ cat main.go
package main

import "fmt"
import "net/http"

func main() {
	fmt.Println("Hello World")

	http.ListenAndServe(":8080", nil)
}
ubuntu@jammy:~$ ~/go/bin/go build -o main-jammy -trimpath ./main.go

Copying the resulting binary to an Ubuntu 20.04 machine yields the following error:

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Dropping -trimpath from the go build call and the binary works.

What did you expect to see?

Invoking main-jammy on an Ubuntu 20.04 machine should yield Hello World and start a webserver.

What did you see instead?

$ ./main-jammy
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./main-jammy)
./main-jammy: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./main-jammy)

Context

Github Actions is currently rolling out Ubuntu 22.04 for its ubuntu-latest tag and as a result some of our releases are becoming invalid with Ubuntu 20.0.

[bcmills]

go version go1.19.4 linux/amd64

Is this a regression from a previous release? (Did it ever work?)

[zeisss]

Previously releases have been build on Ubuntu 20.04 on Github Actions (we are using the ubuntu-latest tag), so I cannot answer that. If I find time tomorrow, I might be able to test this with older releases.

[thanm]

@golang/compiler

[bcmills]

I think -trimpath is something of a red herring. Here's my hypothesis:

• The problem is that your installed glibc has a symbol definition that is newer than the glibc of the machine on which you are deploying the binary.
• When you build without -trimpath, the problem is masked because you link against the precompiled .a files that shipped with go1.19.4, and link against the glibc symbols that happened to be used to build that distribution's cgo dependencies.
• However, when you build with -trimpath, the cache key for the precompiled files no longer matches, and you get a rebuild with your installed glibc.

---

We can bust the cache in other ways, too (such as by setting CGO_CFLAGS), reproducing the same symptom:

$ go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.2.5

$ go1.19.4 build -trimpath -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

$ CGO_CFLAGS='-O2 -g -D__go_unused_variable=1' go1.19.4 build -o main main.go

$ nm -D main | grep pthread_attr_getstacksize
                 U pthread_attr_getstacksize@GLIBC_2.34

[bcmills]

And in Go 1.20, you will see the newer symbols on all builds because we're not shipping precompiled archives at all (#47257). 😅

[bcmills]

The workarounds I can see for your use-case are:

• Build the program on the system with the older glibc (so that it uses the older symbols).
• Set CGO_ENABLED=0 to build a binary that doesn't depend on glibc at all.
• Set various flags (see cmd/go: build: add -static flag #26492) to build a fully-static cgo-enabled binary.

[bcmills]

But, I am left a little bit puzzled about one thing: I don't expect -trimpath to actually affect the cache keys for the standard library: as of CL 380915 we should be building standard-library packages as if -trimpath were always set. 🤔

I'll file a separate issue to look into that.

[bcmills]

I've filed #57346 for the cache-invalidation behavior. Otherwise, I don't think there is anything we can feasibly do for this on the cmd/go side: cgo-enabled Go binaries inherently have many of the same compatibility issues as ordinary C binaries.

[zeisss]

Build the program on the system with the older glibc (so that it uses the older symbols).

Yeah, we will go with that. Since we control most execution environments, this is doable and easiest. I mainly reported this since this was surprising to me and non-obvious from the -trimpath description.

Thanks for looking into this so quickly!