cmd/compile: cross compile for OpenBSD arm64 fails with SIGILL Illegal instruction

[ikmckenz]

Go version

go version go1.22.0 darwin/arm64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/ian/Library/Caches/go-build'
GOENV='/Users/ian/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/ian/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='darwin'
GOPATH='/Users/ian/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/opt/homebrew/Cellar/go/1.22.0/libexec'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/opt/homebrew/Cellar/go/1.22.0/libexec/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.0'
GCCGO='gccgo'
AR='ar'
CC='cc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/gh/8flwtdfn4c526_bx0m7ht8380000gn/T/go-build3231430879=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

Cross compiling a simple hello world program for OpenBSD arm64 fails with SIGILL Illegal instruction.

Hello world code:

package main

import "fmt"

func main() {
    fmt.Println("hello world")
}

Compiled with GOOS=openbsd GOARCH=arm64 go build hello.go from a different machine, and run on an OpenBSD VM running on my MacBook with an M3 Max chip: OpenBSD openbsd-dev.my.domain 7.4 GENERIC.MP#2273 arm64 fails with illegal instruction.

What did you see happen?

Natively building hello world on OpenBSD arm64 works fine, and that machine does have GOOS=openbsd and GOARCH=arm64 defined in the go env, but building the above code with the above command from my darwin/arm64 machine running go1.22.0 as above, and also my linux/amd64 machine running go 1.21.7 produces the illegal instruction error.

GDB on openbsd with the build from linux:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at /usr/lib/go-1.21/src/runtime/rt0_openbsd_arm64.s:15
15	/usr/lib/go-1.21/src/runtime/rt0_openbsd_arm64.s: No such file or directory.

GDB on openbsd from the build from my Mac:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd ()
    at /opt/homebrew/Cellar/go/1.22.0/libexec/src/runtime/rt0_openbsd_arm64.s:15
15	/opt/homebrew/Cellar/go/1.22.0/libexec/src/runtime/rt0_openbsd_arm64.s: No such file or directory.

GDB on openbsd from mac with GOOS=openbsd GOARCH=arm64 go build -trimpath hello.go:

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
15	runtime/rt0_openbsd_arm64.s: No such file or directory.

What did you expect to see?

Note that building the above hello world code into an OpenBSD amd64 binary from both my darwin/arm64 machine and my linux/amd64 machine works and the resulting binary works on a separate OpenBSD amd64 machine I have without issue, so something seems to be wrong with OpenBSD arm64 specifically. I do see openbsd/arm64 in the output of go tool dist list, so I would expect it to work.

[randall77]

Can you print the stack pointer at those failures? It might be incorrectly aligned.

https://stackoverflow.com/questions/63003217/behaviors-of-stack-pointer-in-arm64

[ikmckenz]

Included a couple extra commands of gdb output in case it helps:

(gdb) r
Starting program: /home/ian/hellomactrimpath 

Program received signal SIGILL, Illegal instruction.
_rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
15	runtime/rt0_openbsd_arm64.s: No such file or directory.
(gdb) bt
#0  _rt0_arm64_openbsd () at runtime/rt0_openbsd_arm64.s:15
(gdb) info frame
Stack level 0, frame at 0x68547a1e80:
 pc = 0x73960 in _rt0_arm64_openbsd (runtime/rt0_openbsd_arm64.s:15); 
    saved pc = 0x0
 source language asm.
 Arglist at 0x68547a1e80, args: 
 Locals at 0x68547a1e80, Previous frame's sp is 0x68547a1e80
(gdb) p $sp
$1 = (void *) 0x68547a1e80
(gdb) x/40x $sp
0x68547a1e80:	0x00000001	0x00000000	0x547a1fa0	0x00000068
0x68547a1e90:	0x00000000	0x00000000	0x547a1fbb	0x00000068
0x68547a1ea0:	0x547a1fd8	0x00000068	0x547a1fe4	0x00000068
0x68547a1eb0:	0x547a1ff2	0x00000068	0x547a2001	0x00000068
0x68547a1ec0:	0x547a2014	0x00000068	0x547a201f	0x00000068
0x68547a1ed0:	0x547a2032	0x00000068	0x547a2053	0x00000068
0x68547a1ee0:	0x547a20b2	0x00000068	0x547a20c1	0x00000068
0x68547a1ef0:	0x547a20d5	0x00000068	0x547a20de	0x00000068
0x68547a1f00:	0x547a2110	0x00000068	0x00000000	0x00000000
0x68547a1f10:	0x00000003	0x00000000	0x00010040	0x00000000

[jrick]

this is possibly BTI as it appears that you are running -current. I can't reproduce it on an arm64 machine without BTI (and unfortunately don't have one with it).

[jrick]

Go from ports is patched to mark executables NOBTCFI; something similar is going to be needed in upstream Go as well.

openbsd/ports@71f21f5

[kettenis]

Upstream should really implement support for PAC and/or BTI.

[ikmckenz]

This is a fresh install of 7.4 from install74.img, not -current. Running on a VM on MacOS via UTM (a QEMU/HVF wrapper) with native virtualization, not emulation:

# uname -a
OpenBSD openbsd-dev.my.domain 7.4 GENERIC.MP#2273 arm64

Although separately yes, upstream support for BTI and PAC would be great :)

Is there a way I could use the go build flags to test if this is BTI related?

[jrick]

The ports patch is relevant to the system's native linker (ld.lld), not cross compiling. It is adding a PT_OPENBSD_NOBTCFI section but I'm not sure if this can be done without modifications to cmd/link.

If you update to -current and use ports Go, you could probably test this hypothesis comparing go build -ldflags=-linkmode=external to go build -ldflags=-linkmode=internal. The former would include the PT_OPENBSD_NOBTCFI section and the latter would not.

[sthen]

BTI is used on amd64/arm64 if supported by the cpu in 7.4, no need for -current.

[sthen]

The ports patch is relevant to the system's native linker (ld.lld), not cross compiling. It is adding a PT_OPENBSD_NOBTCFI section but I'm not sure if this can be done without modifications to cmd/link.

If you update to -current and use ports Go, you could probably test this hypothesis comparing go build -ldflags=-linkmode=external to go build -ldflags=-linkmode=internal. The former would include the PT_OPENBSD_NOBTCFI section and the latter would not.

IIRC we have patched for both internal and external linkers.

[jrick]

Ah, yes that true: https://github.com/openbsd/ports/blob/master/lang/go/patches/patch-src_cmd_link_internal_ld_elf_go

So that experiment won't work, both will have the section.

In any case, this seems to be the issue.

[ikmckenz]

I think the issue is my CPU is too new. Reproducing the exact same test on an M1 instead of an M3 Max works without issue, and I now see that only M1 and M2 generation chips are currently supported on the OpenBSD hardware support page.

Curiously, it doesn't seem to be BTI. My M3 Max doesn't show BT in the CPU info in dmesg with the UTM virtualization:

cpu0 at mainbus0 mpidr 0: Unknown, MIDR 0x610f0000
cpu0: 192KB 64b/line 6-way L1 PIPT I-cache, 128KB 64b/line 8-way L1 D-cache
cpu0: 16384KB 128b/line 16-way L2 cache
cpu0: TLBIOS+IRANGE,TS+AXFLAG,FHM,DP,SHA3,RDM,Atomic,CRC32,SHA2+SHA512,SHA1,AES+PMULL,SPECRES,SB,FRINTTS,GPI,LRCPC+LDAPUR,FCMA,JSCVT,API+PAC,DPB,SpecSEI,PAN+ATS1E1+EPAN,LO,HPDS,CSV3,CSV2,DIT

And on the M1 it also doesn't show up:

cpu0 at mainbus0 mpidr 0: Unknown, MIDR 0x610f0000
cpu0: 192KB 64b/line 6-way L1 PIPT I-cache, 128KB 64b/line 8-way L1 D-cache
cpu0: 12288KB 128b/line 12-way L2 cache
cpu0: TLBIOS+IRANGE,TS+AXFLAG,FHM,DP,SHA3,RDM,Atomic,CRC32,SHA2+SHA512,SHA1,AES+PMULL,SPECRES,SB,FRINTTS,GPI,LRCPC+LDAPUR,FCMA,JSCVT,API+PAC,DPB,SpecSEI,PAN+ATS1E1,LO,HPDS,CSV3,CSV2,DIT

Sorry about the false alarm all, I'll close the issue now and wait patiently for M3 support in OpenBSD :)

[jrick]

BTI support was introduced on the M2. It's not unexpected that you wouldn't see this crash on M1.

I don't know why BT is missing from the M3's dmesg. OpenBSD (nor Asahi) does not support booting the platform natively yet.

Perhaps enforcement is still occurring despite it being missing from dmesg.

I would reopen the issue. In the short term, NOBTCFI will work around this, but OpenBSD would prefer that BTI (and IBT on amd64) be properly implemented, and for these instructions to not only be emitted for OpenBSD targets, but all platforms.

[jrick]

NOBTCFI will work around this

Can you build the hello world program from ports Go to confirm this? It should run and you should also see

$ readelf -l ./hello | grep NOBTCF
  OPENBSD_NOBTCF 0x0000000000000000 0x0000000000000000 0x0000000000000000

[ikmckenz]

Confirmed when building with go from ports it runs fine, and that is the exact output from readelf.

[ikmckenz]

Also confirmed the issue is BTI being enforced: I upgraded the VM to the most recent snapshot which includes the recently added code meant to identify this exact scenario, and there it is at the end of the output of kdump clear as day:

 75545 hello    PSIG  SIGILL SIG_DFL code=ILL_BTCFI addr=0x73960 trapno=905969665
 75545 hello    NAMI  "hello.core"

On 7.4 the code was just ILL_ILLOPC.

[gopherbot]

Change https://go.dev/cl/568435 mentions this issue: cmd/link,debug/elf: mark Go binaries with no branch target CFI on openbsd

[ikmckenz]

Just chiming in to confirm that 1e43391 has fixed the issue for me. Compiling go from source after the commit on Linux, and cross building hello world for openbsd/arm64, the resulting binary runs correctly and readelf shows the OPENBSD_NOBTCF section. Thanks!