Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

runtime: unexpected signal during runtime execution in Go 1.17 but not Go 1.16 #46763

Closed
marwan-at-work opened this issue Jun 15, 2021 · 5 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@marwan-at-work
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.17beta1 darwin/amd64

Does this issue reproduce with the latest release?

Yes for 1.17 no for latest stable release (1.16)

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/marwansulaiman/Library/Caches/go-build"
GOENV="/Users/marwansulaiman/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/marwansulaiman/go/pkg/mod"
GONOPROXY=""
GONOSUMDB="marwan.io/*,github.com/marwan-at-work/*,github.com/github/*"
GOOS="darwin"
GOPATH="/Users/marwansulaiman/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17beta1"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/marwansulaiman/marwan/gorme/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/j7/fhzzr84926x491hwp37ltv_r0000gn/T/go-build1384886840=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

The steps are quite reproducible:

  1. Install go1.17 beta for mac.
  2. Run the following commands:
$ mkdir gorme && cd ./gorme
$ go mod init gorme
go: creating new go.mod: module gorme
$ go get github.com/sirupsen/logrus
go get: added github.com/sirupsen/logrus v1.8.1
$ go get github.com/lightstep/lightstep-tracer-go
go get: added github.com/lightstep/lightstep-tracer-go v0.25.0

Add the following main.go file:

package main

import "github.com/sirupsen/logrus"

import (
	_ "github.com/lightstep/lightstep-tracer-go"
)

func main() {
	l := logrus.New()
	l.Info("hello")
}
$ go mod tidy
$ go run -trimpath .
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0xb01dfacedebac1e pc=0x7fff6d88a70a]

runtime stack:
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff6d88a70a
stack: frame={sp:0x7ffeefbff228, fp:0x7ffeefbff278} stack=[0x7ffeefb802c8,0x7ffeefbff330)
0x00007ffeefbff128:  0x01007ffeefbff148  0x0000000000000004 
0x00007ffeefbff138:  0x000000000000001f  0x00007fff6d88a70a 
0x00007ffeefbff148:  0x0b01dfacedebac1e  0x0000000000000001 
0x00007ffeefbff158:  0x0000000004034951 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1f8 
0x00007ffeefbff168:  0x000000000445e6de  0x00007ffeefbff1b0 
0x00007ffeefbff178:  0x0000000004034c08 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000047c4d40 
0x00007ffeefbff188:  0x0000000000000001  0x0000000000000001 
0x00007ffeefbff198:  0x00007ffeefbff1f8  0x0000000004034951 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff1a8:  0x00000000047c4d40  0x00007ffeefbff1e8 
0x00007ffeefbff1b8:  0x0000000004034b90 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff1c8 
0x00007ffeefbff1c8:  0x0000000004034bc0 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000047c4d40 
0x00007ffeefbff1d8:  0x0000000004034951 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1f8 
0x00007ffeefbff1e8:  0x00007ffeefbff218  0x0000000004034951 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff1f8:  0x00007ffeefbff200  0x0000000004034980 <runtime.throw.func1+0x0000000000000000> 
0x00007ffeefbff208:  0x00000000044649ef  0x000000000000002a 
0x00007ffeefbff218:  0x00007ffeefbff268  0x0000000004049f36 <runtime.sigpanic+0x0000000000000396> 
0x00007ffeefbff228: <0x00000000044649ef  0x0000000000000000 
0x00007ffeefbff238:  0x00007ffeefbff2a8  0x0000000004027866 <runtime.(*mheap).allocSpan+0x0000000000000546> 
0x00007ffeefbff248:  0x000000c00023a000  0x0000000000004000 
0x00007ffeefbff258:  0x00007fff00000008  0x000000c0001b0000 
0x00007ffeefbff268:  0x00007ffeefbff2b0 !0x00007fff6d88a70a 
0x00007ffeefbff278: >0x00007ffeefbff2b0  0x0000000004760000 
0x00007ffeefbff288:  0x000000000000043d  0x0000000004101185 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005> 
0x00007ffeefbff298:  0x0000000004065d9f <runtime.syscall+0x000000000000001f>  0x000000c0001af640 
0x00007ffeefbff2a8:  0x000000000406570c <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c0001af610 
0x00007ffeefbff2b8:  0x0000000004063c10 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000002 
0x00007ffeefbff2c8:  0x0000000004005d00 <runtime.cgoCheckWriteBarrier+0x0000000000000020>  0x5d00000000001018 
0x00007ffeefbff2d8:  0x0000000000000000  0x00000000047f7a58 
0x00007ffeefbff2e8:  0x0000000000000a18  0x000000c0000001a0 
0x00007ffeefbff2f8:  0x0000000004061d29 <runtime.systemstack+0x0000000000000049>  0x0000000000000004 
0x00007ffeefbff308:  0x00000000044d2170  0x00000000047c4d40 
0x00007ffeefbff318:  0x00007ffeefbff368  0x0000000004061c25 <runtime.mstart+0x0000000000000005> 
0x00007ffeefbff328:  0x0000000004061bdd <runtime.rt0_go+0x000000000000013d> 
runtime.throw({0x44649ef, 0x0})
        runtime/panic.go:1198 +0x71
runtime: unexpected return pc for runtime.sigpanic called from 0x7fff6d88a70a
stack: frame={sp:0x7ffeefbff228, fp:0x7ffeefbff278} stack=[0x7ffeefb802c8,0x7ffeefbff330)
0x00007ffeefbff128:  0x01007ffeefbff148  0x0000000000000004 
0x00007ffeefbff138:  0x000000000000001f  0x00007fff6d88a70a 
0x00007ffeefbff148:  0x0b01dfacedebac1e  0x0000000000000001 
0x00007ffeefbff158:  0x0000000004034951 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1f8 
0x00007ffeefbff168:  0x000000000445e6de  0x00007ffeefbff1b0 
0x00007ffeefbff178:  0x0000000004034c08 <runtime.fatalthrow.func1+0x0000000000000048>  0x00000000047c4d40 
0x00007ffeefbff188:  0x0000000000000001  0x0000000000000001 
0x00007ffeefbff198:  0x00007ffeefbff1f8  0x0000000004034951 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff1a8:  0x00000000047c4d40  0x00007ffeefbff1e8 
0x00007ffeefbff1b8:  0x0000000004034b90 <runtime.fatalthrow+0x0000000000000050>  0x00007ffeefbff1c8 
0x00007ffeefbff1c8:  0x0000000004034bc0 <runtime.fatalthrow.func1+0x0000000000000000>  0x00000000047c4d40 
0x00007ffeefbff1d8:  0x0000000004034951 <runtime.throw+0x0000000000000071>  0x00007ffeefbff1f8 
0x00007ffeefbff1e8:  0x00007ffeefbff218  0x0000000004034951 <runtime.throw+0x0000000000000071> 
0x00007ffeefbff1f8:  0x00007ffeefbff200  0x0000000004034980 <runtime.throw.func1+0x0000000000000000> 
0x00007ffeefbff208:  0x00000000044649ef  0x000000000000002a 
0x00007ffeefbff218:  0x00007ffeefbff268  0x0000000004049f36 <runtime.sigpanic+0x0000000000000396> 
0x00007ffeefbff228: <0x00000000044649ef  0x0000000000000000 
0x00007ffeefbff238:  0x00007ffeefbff2a8  0x0000000004027866 <runtime.(*mheap).allocSpan+0x0000000000000546> 
0x00007ffeefbff248:  0x000000c00023a000  0x0000000000004000 
0x00007ffeefbff258:  0x00007fff00000008  0x000000c0001b0000 
0x00007ffeefbff268:  0x00007ffeefbff2b0 !0x00007fff6d88a70a 
0x00007ffeefbff278: >0x00007ffeefbff2b0  0x0000000004760000 
0x00007ffeefbff288:  0x000000000000043d  0x0000000004101185 <golang.org/x/sys/unix.libc_ioctl_trampoline+0x0000000000000005> 
0x00007ffeefbff298:  0x0000000004065d9f <runtime.syscall+0x000000000000001f>  0x000000c0001af640 
0x00007ffeefbff2a8:  0x000000000406570c <runtime.nanotime_trampoline+0x000000000000000c>  0x000000c0001af610 
0x00007ffeefbff2b8:  0x0000000004063c10 <runtime.asmcgocall+0x0000000000000070>  0x0000000000000002 
0x00007ffeefbff2c8:  0x0000000004005d00 <runtime.cgoCheckWriteBarrier+0x0000000000000020>  0x5d00000000001018 
0x00007ffeefbff2d8:  0x0000000000000000  0x00000000047f7a58 
0x00007ffeefbff2e8:  0x0000000000000a18  0x000000c0000001a0 
0x00007ffeefbff2f8:  0x0000000004061d29 <runtime.systemstack+0x0000000000000049>  0x0000000000000004 
0x00007ffeefbff308:  0x00000000044d2170  0x00000000047c4d40 
0x00007ffeefbff318:  0x00007ffeefbff368  0x0000000004061c25 <runtime.mstart+0x0000000000000005> 
0x00007ffeefbff328:  0x0000000004061bdd <runtime.rt0_go+0x000000000000013d> 
runtime.sigpanic()
        runtime/signal_unix.go:719 +0x396

goroutine 1 [syscall]:
syscall.syscall(0x4101180, 0x2, 0x40487413, 0xc0001af6d0)
        runtime/sys_darwin.go:22 +0x3b fp=0xc0001af640 sp=0xc0001af620 pc=0x40609bb
syscall.syscall(0x90f0f67d3201c856, 0x7de8cfa85ade3600, 0xb9d8eea6e34af808, 0xd822576cb22b7f61)
        <autogenerated>:1 +0x26 fp=0xc0001af688 sp=0xc0001af640 pc=0x4066566
golang.org/x/sys/unix.ioctl(0x3530ae4f63bfad50, 0x786493e590fecfac, 0x2154961679b0d)
        golang.org/x/sys@v0.0.0-20210217105451-b926d437f341/unix/zsyscall_darwin_amd64.go:689 +0x39 fp=0xc0001af6b8 sp=0xc0001af688 pc=0x4100a39
golang.org/x/sys/unix.IoctlGetTermios(...)
        golang.org/x/sys@v0.0.0-20210217105451-b926d437f341/unix/ioctl.go:72
github.com/sirupsen/logrus.isTerminal(0x87296cca2bde7aa)
        github.com/sirupsen/logrus@v1.8.1/terminal_check_bsd.go:11 +0x50 fp=0xc0001af728 sp=0xc0001af6b8 pc=0x410af30
github.com/sirupsen/logrus.checkIfTerminal({0x44d6820, 0xc000198010})
        github.com/sirupsen/logrus@v1.8.1/terminal_check_notappengine.go:13 +0x75 fp=0xc0001af740 sp=0xc0001af728 pc=0x410afd5
github.com/sirupsen/logrus.(*TextFormatter).init(0xc00019a6c0, 0x8)
        github.com/sirupsen/logrus@v1.8.1/text_formatter.go:106 +0x34 fp=0xc0001af7c0 sp=0xc0001af740 pc=0x410b0b4
github.com/sirupsen/logrus.(*TextFormatter).Format.func1()
        github.com/sirupsen/logrus@v1.8.1/text_formatter.go:196 +0x25 fp=0xc0001af7e0 sp=0xc0001af7c0 pc=0x410c765
sync.(*Once).doSlow(0x44deab0, 0xc0001901c8)
        sync/once.go:68 +0xd2 fp=0xc0001af848 sp=0xc0001af7e0 pc=0x4078312
sync.(*Once).Do(...)
        sync/once.go:59
github.com/sirupsen/logrus.(*TextFormatter).Format(0xc00019a6c0, 0xc00021a1c0)
        github.com/sirupsen/logrus@v1.8.1/text_formatter.go:196 +0xebe fp=0xc0001afdb8 sp=0xc0001af848 pc=0x410c11e
github.com/sirupsen/logrus.(*Entry).write(0xc00021a1c0)
        github.com/sirupsen/logrus@v1.8.1/entry.go:279 +0x56 fp=0xc0001afe48 sp=0xc0001afdb8 pc=0x41063f6
github.com/sirupsen/logrus.(*Entry).log(0xc0001aff60, 0x4, {0xc00019c930, 0x5})
        github.com/sirupsen/logrus@v1.8.1/entry.go:251 +0x219 fp=0xc0001afeb8 sp=0xc0001afe48 pc=0x4105ff9
github.com/sirupsen/logrus.(*Entry).Log(0xc00021a150, 0x4, {0xc0001aff60, 0x70, 0x4431420})
        github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f fp=0xc0001afee8 sp=0xc0001afeb8 pc=0x410664f
github.com/sirupsen/logrus.(*Logger).Log(0xc00021a0e0, 0x4, {0xc0001aff60, 0x1, 0x1})
        github.com/sirupsen/logrus@v1.8.1/logger.go:198 +0x65 fp=0xc0001aff28 sp=0xc0001afee8 pc=0x4109265
github.com/sirupsen/logrus.(*Logger).Info(...)
        github.com/sirupsen/logrus@v1.8.1/logger.go:220
main.main()
        gorme/main.go:11 +0xfe fp=0xc0001aff80 sp=0xc0001aff28 pc=0x43946be
runtime.main()
        runtime/proc.go:255 +0x227 fp=0xc0001affe0 sp=0xc0001aff80 pc=0x4036fe7
runtime.goexit()
        runtime/asm_amd64.s

What did you expect to see?

In Go 1.16 things work fine:

$ docker run --rm -it -v `pwd`:/app golang:1.16
root@0341e88de5ea:/go# cd /app
root@0341e88de5ea:/app# go run -trimpath .
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/lightstep/lightstep-tracer-go v0.25.0
go: downloading golang.org/x/sys v0.0.0-20210217105451-b926d437f341
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20210210170715-a8dfcb80d3a7
go: downloading github.com/opentracing/opentracing-go v1.0.2
go: downloading google.golang.org/grpc v1.21.0
go: downloading github.com/shirou/gopsutil/v3 v3.21.2
go: downloading golang.org/x/net v0.0.0-20201021035429-f5854403a974
go: downloading google.golang.org/genproto v0.0.0-20190530194941-fb225487d101
go: downloading github.com/tklauser/go-sysconf v0.3.4
go: downloading github.com/golang/protobuf v1.3.1
go: downloading github.com/tklauser/numcpus v0.2.1
go: downloading golang.org/x/text v0.3.3
INFO[0000] hello   

What did you see instead?

The segment violation mentioned above.

Note that only when I added the lightstep tracer above did the runtime panic occur. It is probably due to enforcement of an underlying dependency namely golang.org/x/sys.

If I upgrade the underlying dependency (though I've never directly used it), it actually ends up working. To be specific:

This works: golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1
This does not work: golang.org/x/sys v0.0.0-20210217105451-b926d437f341 // indirect

However, in 1.16, both work fine. Which is why I think this is more of a Go bug rather than a library bug.

Thanks!

@davecheney
Copy link
Contributor

The go 1.16 example is using linux inside docker. The 1.17 example looks like it was running on Darwin. Is it possible to eliminate Darwin as a factor?

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 15, 2021
@marwan-at-work
Copy link
Contributor Author

@davecheney good call. It looks like this is only reproducible in Macs:

$ docker run --rm -it -v `pwd`:/app golang:1.17beta1
$ cd /app && go run .
go: downloading github.com/sirupsen/logrus v1.8.1
go: downloading github.com/lightstep/lightstep-tracer-go v0.25.0
go: downloading golang.org/x/sys v0.0.0-20210217105451-b926d437f341
go: downloading github.com/gogo/protobuf v1.3.2
go: downloading github.com/lightstep/lightstep-tracer-common/golang/gogo v0.0.0-20210210170715-a8dfcb80d3a7
go: downloading github.com/opentracing/opentracing-go v1.0.2
go: downloading google.golang.org/grpc v1.21.0
go: downloading github.com/shirou/gopsutil/v3 v3.21.2
go: downloading golang.org/x/net v0.0.0-20201021035429-f5854403a974
go: downloading google.golang.org/genproto v0.0.0-20190530194941-fb225487d101
go: downloading github.com/tklauser/go-sysconf v0.3.4
go: downloading github.com/golang/protobuf v1.3.1
go: downloading github.com/tklauser/numcpus v0.2.1
go: downloading golang.org/x/text v0.3.3
INFO[0000] hello

Do note, however, that this is still a 1.17 bug because Go 1.16 on macs still works with the above program. I was able to reproduce this by doing the following:

$ docker run --rm -it -v `pwd`:/app golang:1.16
$ cd /app
$ GOOS=darwin go build
$ Ctrl+C # to exit container and run volume'd mac binary
$ ./main
INFO[0000] hello

Plus this is an extraction of an internal repository and I had been using Go1.16 to build it for a while and the error did not occur.

Thanks!

@mark-rushakoff
Copy link
Contributor

@marwan-at-work the golang.org/x/sys/unix.libc_ioctl_trampoline error looks like what was fixed as part of #45702. The fix on your end should be to update golang.org/x/sys to latest (it looks like you are on a version before the fix at golang/sys@a76c4d0).

@marwan-at-work
Copy link
Contributor Author

@mark-rushakoff that sounds great. However, I still find it odd that the same x/sys module@version works in Go1.16 but not in 1.17. Typically, upgrading a minor Go version means your program should continue to work. In this case, someone in my situation will need to realize that upgrading from Go 1.16 to Go 1.17 is a bit of a breaking change because they will also need to upgrade an underlying dependency which may not be an option for them.

Luckily for me, I'm happy to upgrade x/sys but just thought I'd point it out.

Thanks again for the quick responses ✌🏼

@cherrymui
Copy link
Member

The previous version of x/sys use some assembly code in a subtle way that happens to work with Go 1.16 but not with 1.17 (this is not covered by the compatibility guarantee). As commented previously, updating to a newer version of x/sys works. Closing. Thanks.

bouk added a commit to bouk/devd that referenced this issue Aug 19, 2021
bouk added a commit to bouk/yubikey-agent that referenced this issue Aug 19, 2021
xyproto pushed a commit to xyproto/algernon that referenced this issue Aug 20, 2021
semihalev added a commit to semihalev/sdns that referenced this issue Aug 20, 2021
See: golang/go#46763

Co-authored-by: Yasar Alev <semihalev@gmail.com>
AlekSi added a commit to AlekSi/Xray-core that referenced this issue Aug 20, 2021
AlekSi added a commit to AlekSi/orgalorg that referenced this issue Aug 20, 2021
bouk added a commit to bouk/CovenantSQL that referenced this issue Aug 22, 2021
aeneasr pushed a commit to ory/hydra that referenced this issue Aug 23, 2021
FiloSottile pushed a commit to FiloSottile/yubikey-agent that referenced this issue Aug 28, 2021
meowgorithm added a commit to meowgorithm/sqlite-tui that referenced this issue Sep 2, 2021
meowgorithm added a commit to meowgorithm/sqlite-tui that referenced this issue Sep 2, 2021
meowgorithm added a commit to meowgorithm/sqlite-tui that referenced this issue Sep 2, 2021
meowgorithm added a commit to meowgorithm/sqlite-tui that referenced this issue Sep 2, 2021
zmwangx pushed a commit to fanaticscripter/EggContractor that referenced this issue Sep 22, 2021
Mainly to update x/sys so that the program doesn't segfault when built with go
1.17. See golang/go#46763.
ndhanushkodi added a commit to hashicorp/consul-k8s that referenced this issue Oct 14, 2021
See golang/go#46763. The CLI would error at
runtime after being compiled with go 1.17, unless the x/sys dependency
is bumped.
ndhanushkodi added a commit to hashicorp/consul-k8s that referenced this issue Oct 14, 2021
See golang/go#46763. The CLI would error at
runtime after being compiled with go 1.17, unless the x/sys dependency
is bumped.
muesli pushed a commit to charmbracelet/charm that referenced this issue Dec 6, 2021
@golang golang locked and limited conversation to collaborators Jun 15, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants