runtime: fatal error: unknown caller pc when uprobes are attached #27077
Comments
|
cc @aclements Speculation, but one possible reason reading certs might matter is that it makes a cgo call. |
josharian
changed the title
|
This definitely looks like uprobes is installing a return trampoline on the stack to catch function return, which is going to mess up Go stack unwinding no matter what. I don't see why this would require a cgo call. What happens if you simply have something like func x() {
runtime.GC()
}and set a return uprobe on (or apply funclatency to) |
|
Yeah, I just poked through the uretprobe implementation and it works by allocating a page in user space, putting an INT $3 instruction on it, and then overwriting the return address on the stack to return to the INT $3. It's not clear to me how the runtime (or anything that unwinds the stack) could account for this. |
|
@aclements I'm not sure I understand the issue here. |
|
The only difference is that Go depends on its ability to unwind stacks for GC and stack growth. I assert that uretprobes would break stack unwinding in any language, and regardless of calling convention. I'm actually really curious how it interacts with C++ exception handling; I suspect uretprobes breaks it. uretprobes clobbers critical unwinding information and, as far as I can tell, doesn't provide a way to get it back. However, I would love to be proved wrong, since I know how powerful uprobes can be. |
|
related discussions: |
|
Thanks for the references, @sillyousu. Those confirm my suspicions that, unfortunately, there's really nothing we can do to work around uretprobes effectively corrupting the stack. Since there's nothing we can do, and this isn't really a Go bug, I'm going to close this issue. If uretprobes one day exposes enough information to recover the clobbered return address in user space, we can revisit this issue and might be able to work around it. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
This is the latest release.
What operating system and processor architecture are you using (
go env)?What did you do?
Compiled:
Then attach uprobe with bcc tools:
Then run the program:
This doesn't happen with simple calls and I wasn't unable to write a simpler reproduction that doesn't go into guts of certificate validation. With detached uprobes there's no error.
What did you expect to see?
Program successfully terminates, uprobes do their business.
What did you see instead?
Program aborts, uprobes never see anything.
The text was updated successfully, but these errors were encountered: