runtime/cgo: unrecognized relocation with binaries shipped in go1.12.2

[jayconrod]

What version of Go are you using (go version)?

go1.12.2

This specifically affects the binary distribution downloaded from golang.org. Bootstrapping 1.12.2 using 1.12.1 did not have this problem.

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/bazel/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/bazel/go"
GOPROXY=""
GORACE=""
GOROOT="/home/bazel/go1.12.2"
GOTMPDIR=""
GOTOOLDIR="/home/bazel/go1.12.2/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build845022667=/tmp/go-build -gno-record-gcc-switches"

This error is most easily reproduced with the gcr.io/bazel-untrusted/ubuntu1404:java8 docker image that Bazel uses for testing. Sorry, it's pretty big.

The important part of this is that it's on ubuntu14.04, which has an older linker.

bazel@269f5850547f:~$ /usr/bin/ld -v
GNU ld (GNU Binutils for Ubuntu) 2.24
bazel@269f5850547f:~$ /usr/bin/ld.gold -v
GNU gold (GNU Binutils for Ubuntu 2.24) 1.11
bazel@269f5850547f:~$ /usr/bin/gcc -v
Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.8/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.8.4-2ubuntu1~14.04.4' --with-bugurl=file:///usr/share/doc/gcc-4.8/README.Bugs --enable-languages=c,c++,java,go,d,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.8 --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.8 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --disable-libmudflap --enable-plugin --with-system-zlib --disable-browser-plugin --enable-java-awt=gtk --enable-gtk-cairo --with-java-home=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64/jre --enable-java-home --with-jvm-root-dir=/usr/lib/jvm/java-1.5.0-gcj-4.8-amd64 --with-jvm-jar-dir=/usr/lib/jvm-exports/java-1.5.0-gcj-4.8-amd64 --with-arch-directory=amd64 --with-ecj-jar=/usr/share/java/eclipse-ecj.jar --enable-objc-gc --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.4) 

What did you do?

Download go1.12.2

Try to build the program below:

package main

// const int x = 42;
import "C"

import "fmt"

func main() {
  fmt.Println(int(C.x))
}

~/go1.12.2/bin/go build hello.go

What did you expect to see?

Successful build.

What did you see instead?

# command-line-arguments
/home/bazel/go1.12.2/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /tmp/go-link-648962525/000005.o: unrecognized relocation (0x2a) in section `.text'
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status

I'm seeing similar error when Bazel links binaries with cgo code.

external/go_sdk/pkg/tool/linux_amd64/link: running /usr/bin/gcc failed: exit status /usr/bin/ld.gold: error: /tmp/go-link-028957856/000005.o: unsupported reloc 42 against global symbol stderr
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000006.o: unsupported reloc 42 against global symbol stderr
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000007.o: unsupported reloc 42 against global symbol x_cgo_threadentry
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000007.o: unsupported reloc 42 against global symbol x_cgo_inittls
/usr/bin/ld.gold: error: /tmp/go-link-028957856/000012.o: unsupported reloc 42 against global symbol stderr
gcc_fatalf.c:17: error: unsupported reloc 42
gcc_libinit.c:29: error: unsupported reloc 42
gcc_linux_amd64.c:86: error: unsupported reloc 42
gcc_linux_amd64.c:52: error: unsupported reloc 42
gcc_util.c:18: error: unsupported reloc 42
collect2: error: ld returned 1 exit status

Other thoughts

@ianlancetaylor told me that reloc 42 is R_X86_64_REX_GOTPCRELX. Support for this was added to gold in 2015, so it's presumably not supported by ubuntu14.04. Support was added to the Go linker for #13114. The Go toolchain should not be emitting these relocs. We may be incorporating .o files produced by a C compiler that emits these on the system that produces the binary distribution.

I was able to work around this by bootstrapping the Go 1.12.2 toolchain locally. I think the C compiler installed on this image does not emit these relocs.

[jayconrod]

cc @ianlancetaylor @dmitshur

[ianlancetaylor]

The relocation appears in runtime/cgo for C code that refers to global variables.

The 1.12.2 binary distribution must have been built with a newer version of GCC than the 1.12.1 binary distribution. The only reasonable fix is to build it with an older version of GCC.

I'm not sure whether we should do anything about this. As a general rule, if you want the binary distribution to run on release N, you need to build it on release N, not a later release. This is one example of that, but there are many others. I don't remember whether we've ever documented anything about what systems the binary distributions run on.

CC @bradfitz @andybons

[bradfitz]

We updated our linux-amd64 builder image from Debian Jessie to Stretch. That's surely the cause. We obviously didn't consider that it'd have any effect on release binaries. We also don't version our builder images and our release automation doesn't use different builders as a function of branch either. Maybe it should. We never thought about it.

[bradfitz]

Actually we do version our builder images a bit at a broad level. But we do update a number of them "in-place" without changing their names and we definitely don't select release builders as a function of branch.

[sbinet]

just to note this is also the case for the go1.11.7 builds:

• https://travis-ci.org/go-hep/hep/jobs/516531718

[karalabe]

Whoah, all our Travis builds blew up:

• Go 1.11: https://travis-ci.org/ethereum/go-ethereum/jobs/516456569#L954
• Go 1.12: https://travis-ci.org/ethereum/go-ethereum/jobs/516456570#L946

[jayconrod]

For Travis CI errors, if you're willing to upgrade to Ubuntu 16.04, you can do so by changing dist: trusty to dist: xenial in your .travis.yml file.

[aronatkins]

On Ubuntu 14.04:

$ ld -v
GNU ld (GNU Binutils for Ubuntu) 2.24

Issue #31336 is similar. We see this failure on Ubuntu 14.04 and the #31336 failure on CentOS 6 build hosts.

[ianlancetaylor]

This was reported again on #31336. We may need to plan to fix this.

[gopherbot]

Change https://golang.org/cl/171121 mentions this issue: dashboard, cmd/release: use Jessie for release builds

[andybons]

@gopherbot please backport to 1.11 and 1.12; the fix doesn't need to be cherry-picked, but it will be used to track the issue in the milestone.

[gopherbot]

Backport issue(s) opened: #31346 (for 1.11), #31347 (for 1.12).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[andybons]

1.12.3 and 1.11.8 have been built using Jessie and released. This should resolve the issue. Please let us know if there are any other problems and sorry for the trouble.

Thanks.

[mwhudson]

FWIW, this particular issue can be mitigated by passing "-Wa,-mrelax-relocations=no" when compiling. But if you do that, unless you set CGO_FLAGS to this every time you build, the new (ish) code in the go tool considers the distributed runtime/cgo.a stale and doesn't use it. In fact, given the build cache I'm not sure there is any real need to distribute runtime/cgo.a at all -- I stopped for my snap and as far as I can tell noone has noticed...