runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 #37127

tonyghita · 2020-02-07T21:51:57Z

What version of Go are you using (`go version`)?

$ go version
go1.14rc1 linux/amd64

Does this issue reproduce with the latest release?

Indeed

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build146257936=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Built my application with the latest go1.14rc1 image on the official Docker registry, copied it to an alpine-3.8 based image and ran traffic through it.

What did you expect to see?

No crashes. This application has not exhibited this fatal error before go1.14rc1 (currently on go1.13.7).

What did you see instead?

runtime: invalid pc-encoded table f=runtime.recv pc=0x40625f targetpc=0x40625f tab=[0/0]0x0
value=0 until pc=0x4060c7
value=48 until pc=0x40615f
value=0 until pc=0x406160
value=48 until pc=0x406255
value=0 until pc=0x40625f
fatal error: invalid runtime symbol table
goroutine 0 [idle]:
runtime: unexpected return pc for runtime.sigreturn called from 0x7
stack: frame={sp:0xc29758b380, fp:0xc29758b388} stack=[0xc297584000,0xc29758c000)
000000c29758b280: 000000c297582000 0000000000000000
000000c29758b290: 000000c29758b4b0 000000c29758b380
000000c29758b2a0: 000000c29758b318 0000000000449e69 <runtime.sigtrampgo+409>
000000c29758b2b0: 000000c20000001b 000000c29758b4b0
000000c29758b2c0: 000000c29758b380 000000c297582180
000000c29758b2d0: 0000000000000000 0000000000000000
000000c29758b2e0: 0000000000000000 0000000000000000
000000c29758b2f0: 0000000000000000 0000000000000000
000000c29758b300: 000000c297582180 000000c29758b4b0
000000c29758b310: 000000c29758b380 000000c29758b370
000000c29758b320: 00000000004687e3 <runtime.sigtramp+67> 000000000000001b
000000c29758b330: 000000c29758b4b0 000000c29758b380
000000c29758b340: 0000000000000002 0000000000000002
000000c29758b350: 000000c297593e2c 00000000135c9af0
000000c29758b360: 000000c29758b370 00007fffea5da080
000000c29758b370: 000000c297593e10 00000000004688d0 <runtime.sigreturn+0>
000000c29758b380: <0000000000000007 >0000000000000000
000000c29758b390: 000000c297584000 0000000000000000
000000c29758b3a0: 0000000000008000 00007fffea5db000
000000c29758b3b0: 00065dca43412f3f 00081e2d2dc1e26b
000000c29758b3c0: 0000000000000001 00000000135c9af0
000000c29758b3d0: 000000c297593e2c 0000000000000002
000000c29758b3e0: 0000000000000002 000000c297593e2c
000000c29758b3f0: 0000000000000048 000000c297593e10
000000c29758b400: 00007fffea5da080 0000000000043e86
000000c29758b410: d77c8ba3287f35d6 00000000ffffffff
000000c29758b420: 000000c297593e10 00007fffea5dd9e7
000000c29758b430: 0000000000000a83 002b000000000033
000000c29758b440: 0000000000000000 0000000000000000
000000c29758b450: 0000000000000000 0000000000000000
000000c29758b460: 000000c29758b540 0000000000000000
000000c29758b470: 0000000000000000 0000000000000000
000000c29758b480: 0000000000000000
runtime.throw(0x1faacbd, 0x1c)
/usr/local/go/src/runtime/panic.go:1112 +0x72
runtime.pcvalue(0x25434a8, 0x357aaa0, 0x1027b5, 0x40625f, 0xc29758adc8, 0x2eb7101, 0x0)
/usr/local/go/src/runtime/symtab.go:726 +0x53a
runtime.funcspdelta(0x25434a8, 0x357aaa0, 0x40625f, 0xc29758adc8, 0xc200000000)
/usr/local/go/src/runtime/symtab.go:780 +0x5f
runtime.gentraceback(0x40625f, 0xc297593e98, 0x0, 0xc297582180, 0x0, 0xc29758b020, 0x40, 0x0, 0x0, 0x6, ...)
/usr/local/go/src/runtime/traceback.go:220 +0x15ae
runtime.sigprof(0x7fffea5dd9e7, 0xc297593e10, 0x0, 0xc297582180, 0xc297580000)
/usr/local/go/src/runtime/proc.go:3897 +0x371
runtime.sighandler(0xc20000001b, 0xc29758b4b0, 0xc29758b380, 0xc297582180)
/usr/local/go/src/runtime/signal_unix.go:522 +0x787
runtime.sigtrampgo(0x1b, 0xc29758b4b0, 0xc29758b380)
/usr/local/go/src/runtime/signal_unix.go:444 +0x199
runtime.sigtramp(0x7, 0x0, 0xc297584000, 0x0, 0x8000, 0x7fffea5db000, 0x65dca43412f3f, 0x81e2d2dc1e26b, 0x1, 0x135c9af0, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:389 +0x43
runtime: unexpected return pc for runtime.sigreturn called from 0x7
stack: frame={sp:0xc29758b380, fp:0xc29758b388} stack=[0xc297584000,0xc29758c000)
000000c29758b280: 000000c297582000 0000000000000000
000000c29758b290: 000000c29758b4b0 000000c29758b380
000000c29758b2a0: 000000c29758b318 0000000000449e69 <runtime.sigtrampgo+409>
000000c29758b2b0: 000000c20000001b 000000c29758b4b0
000000c29758b2c0: 000000c29758b380 000000c297582180
000000c29758b2d0: 0000000000000000 0000000000000000
000000c29758b2e0: 0000000000000000 0000000000000000
000000c29758b2f0: 0000000000000000 0000000000000000
000000c29758b300: 000000c297582180 000000c29758b4b0
000000c29758b310: 000000c29758b380 000000c29758b370
000000c29758b320: 00000000004687e3 <runtime.sigtramp+67> 000000000000001b
000000c29758b330: 000000c29758b4b0 000000c29758b380
000000c29758b340: 0000000000000002 0000000000000002
000000c29758b350: 000000c297593e2c 00000000135c9af0
000000c29758b360: 000000c29758b370 00007fffea5da080
000000c29758b370: 000000c297593e10 00000000004688d0 <runtime.sigreturn+0>
000000c29758b380: <0000000000000007 >0000000000000000
000000c29758b390: 000000c297584000 0000000000000000
000000c29758b3a0: 0000000000008000 00007fffea5db000
000000c29758b3b0: 00065dca43412f3f 00081e2d2dc1e26b
000000c29758b3c0: 0000000000000001 00000000135c9af0
000000c29758b3d0: 000000c297593e2c 0000000000000002
000000c29758b3e0: 0000000000000002 000000c297593e2c
000000c29758b3f0: 0000000000000048 000000c297593e10
000000c29758b400: 00007fffea5da080 0000000000043e86
000000c29758b410: d77c8ba3287f35d6 00000000ffffffff
000000c29758b420: 000000c297593e10 00007fffea5dd9e7
000000c29758b430: 0000000000000a83 002b000000000033
000000c29758b440: 0000000000000000 0000000000000000
000000c29758b450: 0000000000000000 0000000000000000
000000c29758b460: 000000c29758b540 0000000000000000
000000c29758b470: 0000000000000000 0000000000000000
000000c29758b480: 0000000000000000
runtime.sigreturn(0x0, 0xc297584000, 0x0, 0x8000, 0x7fffea5db000, 0x65dca43412f3f, 0x81e2d2dc1e26b, 0x1, 0x135c9af0, 0xc297593e2c, ...)
/usr/local/go/src/runtime/sys_linux_amd64.s:481

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2020-02-07T22:35:53Z

CC @aclements @randall77

Is there a way that we can reproduce the problem ourselves?

tonyghita · 2020-02-07T22:40:26Z

In ~25 minutes of running production traffic at just above 700 request per second per application, this occurred 12 times across the fleet (different instances each time). Each fatal error occurred about 1 to 4 minutes after the application started and received traffic. The application is a non-trivial HTTP API (GraphQL) gateway that aggregates responses from other services.

Each time the specific runtime: invalid pc-encoded table value of f is different, but messages fatal error: invalid runtime symbol table and runtime: unexpected return pc for runtime.sigreturn called from 0x7 appear in every case.

Open to ideas for reproducing this case more minimally. Wild uneducated guess, but I saw a similar issue (#27540) call out profiling as a possible cause, which we do in this application (pprof on an interval with block and mutex profiles enabled). It has not been an issue until go1.14rc1 though.

laboger · 2020-02-11T17:24:04Z

We see this error in testing Openshift when compiled with go1.14rc1 on ppc64le. By default this testing is done with the -race option on, and that results in a few dozen errors about unsafe pointer arithmetic, since -race now turns on checkptr testing.

If I then run the tests with checkptr=0 along with -race, then I see the same with the invalid pc-encoded symbol table as above.

I'm still trying to isolate the conditions which cause this consistently because from run to run the failures are sometimes different. So far it has only happens when running all the tests, if I try to just run a test that previously failed by itself it does not fail. If I test without -race then different errors occur.

cherrymui · 2020-02-11T19:51:45Z

I think this may be related to CL https://go-review.googlesource.com/c/go/+/212079 . The code saving vdsoPC in walltime1/nanotime1 assumes the function has no frame
https://go-review.googlesource.com/c/go/+/212079/3/src/runtime/sys_linux_amd64.s#224
Now the functions do have frame, so the offset needs adjustment.

That said, if my assumption is right, the PPC64 failure is probably a different one.

gopherbot · 2020-02-11T20:24:40Z

Change https://golang.org/cl/219118 mentions this issue: runtime: correct caller PC/SP offsets in walltime1/nanotime1

dmitshur · 2020-02-13T20:22:01Z

The text in commit message of CL 219118 included:

May fix #37127.

Unfortunately, GitHub parses that as "Fix[es] #37127." and automatically closed this issue. I don't think that was intended, so re-opening. /cc @cherrymui

cherrymui · 2020-02-13T20:33:59Z

Given that the crash is hard to reproduce, I cannot tell for sure whether the problem goes away with this, so "may fix". I guess we can close this and, if it didn't work, reopen it. Or we keep it open and let the reporter confirm.

laboger · 2020-02-13T20:38:42Z

I am still working on trying to narrow down the failure that happens on ppc64le. Should I open a separate issue for that?

I believe it began when the page allocator changed in early November but still trying to verify and find a smaller reproducer.

cherrymui · 2020-02-13T20:41:01Z

@laboger yeah, I think it's better to open a new one. Quick question: do you have profiling turned on? If not, it is clearly a different problem.

laboger · 2020-02-13T20:50:23Z

@cherrymui It is not happening with profiling but fails with -race -d=checkptr=0. If I turn off -race it gets a different error. I'll open a new issue. It is the same error output about the symbol table.

tonyghita · 2020-02-13T21:01:06Z

I'm eager to try your fix @cherrymui. Will this be released in something like an RC2?

cherrymui · 2020-02-13T21:17:44Z

Yeah, there might be one ~next week or so. You could also check out Go tip, which is very close to 1.14rc1, with just a small number of fixes. Thanks!

aclements · 2020-02-13T21:43:39Z

Removing release-blocker, since we think this is probably fixed. If it turns out not to be, we can continue working on this and perhaps issue a fix in a point release.

tonyghita · 2020-02-19T22:06:55Z

Hey all, just tested the changes in tip and wanted to confirm that I no longer see the problem on linux/amd64 with my workload. Good find and thank you!

dmitshur · 2020-02-19T22:13:22Z

Thanks for confirming @tonyghita!

I believe this issue is resolved then. It was waiting on feedback from you, the original reporter. I'll close it because there's nothing left to do. If there's anything else, please let us know!

cherrymui · 2020-02-19T22:17:15Z

Thanks @tonyghita

tonyghita changed the title ~~runtime: go1.14rc1 fatal error: invalid runtime symbol table~~ runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 Feb 7, 2020

ianlancetaylor added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Feb 7, 2020

ianlancetaylor added this to the Go1.14 milestone Feb 7, 2020

aclements added the release-blocker label Feb 13, 2020

gopherbot closed this as completed in 3eab754 Feb 13, 2020

dmitshur reopened this Feb 13, 2020

dmitshur added the OS-Linux label Feb 13, 2020

aclements removed the release-blocker label Feb 13, 2020

dmitshur added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 14, 2020

dmitshur closed this as completed Feb 19, 2020

golang locked and limited conversation to collaborators Feb 18, 2021

gopherbot added the FrozenDueToAge label Feb 18, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 #37127

runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 #37127

tonyghita commented Feb 7, 2020 •

edited

Loading

ianlancetaylor commented Feb 7, 2020

tonyghita commented Feb 7, 2020 •

edited

Loading

laboger commented Feb 11, 2020

cherrymui commented Feb 11, 2020

gopherbot commented Feb 11, 2020

dmitshur commented Feb 13, 2020 •

edited

Loading

cherrymui commented Feb 13, 2020

laboger commented Feb 13, 2020

cherrymui commented Feb 13, 2020

laboger commented Feb 13, 2020 •

edited

Loading

tonyghita commented Feb 13, 2020

cherrymui commented Feb 13, 2020

aclements commented Feb 13, 2020

tonyghita commented Feb 19, 2020

dmitshur commented Feb 19, 2020

cherrymui commented Feb 19, 2020

runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 #37127

runtime: go1.14rc1 fatal error: invalid runtime symbol table: runtime: unexpected return pc for runtime.sigreturn called from 0x7 #37127

Comments

tonyghita commented Feb 7, 2020 • edited Loading

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

ianlancetaylor commented Feb 7, 2020

tonyghita commented Feb 7, 2020 • edited Loading

laboger commented Feb 11, 2020

cherrymui commented Feb 11, 2020

gopherbot commented Feb 11, 2020

dmitshur commented Feb 13, 2020 • edited Loading

cherrymui commented Feb 13, 2020

laboger commented Feb 13, 2020

cherrymui commented Feb 13, 2020

laboger commented Feb 13, 2020 • edited Loading

tonyghita commented Feb 13, 2020

cherrymui commented Feb 13, 2020

aclements commented Feb 13, 2020

tonyghita commented Feb 19, 2020

dmitshur commented Feb 19, 2020

cherrymui commented Feb 19, 2020

tonyghita commented Feb 7, 2020 •

edited

Loading

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

tonyghita commented Feb 7, 2020 •

edited

Loading

dmitshur commented Feb 13, 2020 •

edited

Loading

laboger commented Feb 13, 2020 •

edited

Loading