Gotrace

[nyarly]

This touches on #934 - it adds a rough gotrace.py tool (which still would require documentation etc), but handles Go gc compiled code by using stack offsets, and manages Go's Pascal-style strings. I'm still not sure how to handle slices or concrete structs, but it seemed reasonable to present this for comment now that it will work for some cases.

The appropriate comparison is to trace.py - reviewing what's changed, I think it would be possible to merge most of the changes back into trace.py with a few caveats -

• to format Go strings properly, we need to capture the length, and then trim them in Python (the BPF validator doesn't like variable length string copies.) That behavior obviously isn't consistent with C-style null-terminated strings.
• future work should address slices, which also transmit a capacity argument; if anything, they'll need to be formatted differently
• I'm not certain, but I suspect that signatures in the probe definition won't work with Go programs at all.

I think this all could be handled by a flag and conditionals within trace.py

[nyarly]

Does this make sense, @brendangregg @goldshtn I'd like to make progress on this, but I'd like some sense of what y'all think about this direction.

If it were to go back into trace.py, it occurred to me that, basically, it would need an expansion to the trace definition language - since afaict it doesn't make sense to allow signatures for Go gc traces.

[nyarly]

Just a reminder on this PR - @goldshtn @brendangregg

[nyarly]

Renewing my request for review, @goldshtn & @brendangregg - I got totally distracted myself, but I'd like to make progress on this

[bobrik]

FYI: golang/go#27077

[nyarly]

@bobrik I don't follow that issue completely. How does unwinding the stack relate? Is it peculiar to Go?

[bobrik]

I was playing with funclatency and it broke with my test go program. Tracing of uretprobes can get into the same issues.

[nyarly]

Do you understand the issue with unwinding the stack? Are uretprobes just fundamentally incompatible with Go programs - seems to me that they sometimes work, but I'm unclear on when or why. At the least, should we consider if BCC tools can determine if a particular probe will be incompatible?

[bobrik]

I don't have enough knowledge to understand the issue with stack unwinding, sorry. My hope is that Go developers can figure this out.

[nyarly]

@bobrik This comment seems to summarize the whole thing: #1320 (comment)

I'm not entirely sure what to do about this for this PR, which is kind of languishing anyway.

[Julio-Guerra]

What about making BCC tools ABI-aware?

[nyarly]

@Julio-Guerra I'm not sure I understand what you mean?

[Julio-Guerra]

@Julio-Guerra I'm not sure I understand what you mean?

Since, I guess, trace.py expects C conventions that are for example false with Go, one way to merge your work into trace.py would be to abstract this out so that trace.py no longer makes assumptions on the binary interfaces and can therefore be ported to other languages.

[nyarly]

The C conventions are pretty universal in Linux, though, as I understand it. Go's "everything goes on the stack directly" is a significant outlier, which also foils uretprobes, unfortunately. It's natural to want to say "report the 3rd argument when called", but that means something totally different to Go than it does to everything else. It's different enough that the Go trace version needs not only needs to handle it differently, but also present a different user interface.

[brendangregg]

How much delta is there between gotrace and trace? If it's somewhat significant, then I'd go ahead with accepting gotrace into bcc.

I was just rereading Gianluca Borello's experiments with using uprobes on the return points (#1320 (comment)), and that's something that could be added to gotrace. With such a change, it'd definitely be a significant divergence from trace.

[bobek]

This may be of interest -- golang/go#40724 -- they are proposing to switch to register-based calling for go 1.16.