It needs to support retrieving status codes in gRPC client versions prior to v1.40.

[bittercookie]

In the file /bpf/google.golang.org/grpc/client/bpf/probe.bpf.c, the uprobe_ClientConn_Invoke_Returns method reads the return value from the Error.s.s.Code field as the status code.

However, the structure of google.golang.org/grpc/internal/status.Error varies across different versions:

For v1.30.0 - v1.39.1 (link):

type Error struct {
    e *spb.Status
}

For v1.29.0 - v1.29.1 (link):

type Error spb.Status

The structure spb.Status is defined in google.golang.org/genproto/googleapis/rpc/status.Status.

For v1.8.0 - v1.28.1, the returned Error is actually the struct google.golang.org/grpc/status.statusError same as v1.29.0 - v1.29.1 (link):

type statusError spb.Status

Therefore, the logic to read the status code should be implemented according to the different structures.

In offset_results.json, add offset definitions for google.golang.org/grpc/internal/status.Error:

google.golang.org/grpc/internal/status.Error.e
google.golang.org/grpc/internal/status.Error.Code

And for google.golang.org/grpc/status.statusError:

google.golang.org/grpc/status.statusError.Code

In the uprobe_ClientConn_Invoke_Returns method, retrieve the corresponding status code by checking for non-null addresses in Error.s, Error.e, and Error.Code.

[damemi]

@bittercookie thanks for reporting this and for the detailed report, can I just ask what is the behavior you're seeing when this fails? Is the span still present (just without an error/status code) or does the entire probe fail? Thanks

[bittercookie]

otel-go-instrumentation can not work

[nb@localhost tmp]$ sudo OTEL_GO_AUTO_TARGET_EXE=/home/nb/tmp/grpc7-v1.36.0-go-1.21-static OTEL_SERVICE_NAME=grpc_service OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318 ./otel-go-instrumentation
{"time":"2024-10-30T09:17:54.85644044+08:00","level":"INFO","source":{"function":"main.main","file":"/app/cli/main.go","line":103},"msg":"building OpenTelemetry Go instrumentation ...","globalImpl":false,"version":{"Release":"v0.16.0-alpha","Revision":"unknown","Go":{"Version":"go1.23.2","OS":"linux","Arch":"amd64"}}}
{"time":"2024-10-30T09:17:56.859616982+08:00","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/process.(*Analyzer).DiscoverProcessID","file":"/app/internal/pkg/process/discover.go","line":64},"msg":"found process","pid":2475}
{"time":"2024-10-30T09:17:56.869017373+08:00","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/process.remoteAllocate.func1","file":"/app/internal/pkg/process/allocate.go","line":70},"msg":"Detaching from process","pid":2475}
{"time":"2024-10-30T09:17:56.869067996+08:00","level":"INFO","source":{"function":"go.opentelemetry.io/auto.NewInstrumentation","file":"/app/instrumentation.go","line":113},"msg":"target process analysis completed","pid":2475,"go_version":"1.21.13","dependencies":{"github.com/golang/protobuf":"1.4.2","golang.org/x/net":"0.0.0-20190311183353-d8887717615a","golang.org/x/sys":"0.0.0-20190215142949-d0b11bdaac8a","golang.org/x/text":"0.3.0","google.golang.org/genproto":"0.0.0-20200526211855-cb27e3aa2013","google.golang.org/grpc":"1.27.0","google.golang.org/protobuf":"1.25.0","std":"1.21.13"},"total_functions_found":8}
{"time":"2024-10-30T09:17:56.869134972+08:00","level":"INFO","source":{"function":"main.main","file":"/app/cli/main.go","line":134},"msg":"starting instrumentation..."}
{"time":"2024-10-30T09:17:56.86935077+08:00","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load","file":"/app/internal/pkg/instrumentation/manager.go","line":324},"msg":"loading probe","name":{"SpanKind":3,"InstrumentedPkg":"net/http"}}
{"time":"2024-10-30T09:17:56.976383446+08:00","level":"INFO","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load","file":"/app/internal/pkg/instrumentation/manager.go","line":324},"msg":"loading probe","name":{"SpanKind":3,"InstrumentedPkg":"google.golang.org/grpc"}}
{"time":"2024-10-30T09:17:56.97755516+08:00","level":"ERROR","source":{"function":"go.opentelemetry.io/auto/internal/pkg/instrumentation.(*Manager).load","file":"/app/internal/pkg/instrumentation/manager.go","line":327},"msg":"error while loading probes, cleaning up","error":"invalid offset for the field in version: google.golang.org/grpc/internal/status.Error:s (1.27.0)\ninvalid offset for the field in version: google.golang.org/grpc/internal/status.Status:s (1.27.0)","name":{"SpanKind":3,"InstrumentedPkg":"google.golang.org/grpc"}}
{"time":"2024-10-30T09:17:57.12838706+08:00","level":"ERROR","source":{"function":"main.main","file":"/app/cli/main.go","line":136},"msg":"instrumentation crashed","error":"invalid offset for the field in version: google.golang.org/grpc/internal/status.Error:s (1.27.0)\ninvalid offset for the field in version: google.golang.org/grpc/internal/status.Status:s (1.27.0)"}

[bittercookie]

You can obtain the test program from flyserver/grpc7:grpc7-v1.39.0-go-1.21-static:

$ docker run --rm --name grpc7-container flyserver/grpc7:grpc7-v1.39.0-go-1.21-static &
$ docker cp grpc7-container:/usr/bin/grpc7-v1.39.0-go-1.21-static /tmp/grpc7-v1.39.0-go-1.21-static
$ docker stop grpc7-container

Then, start the test program with /tmp/grpc7-v1.39.0-go-1.21-static and use curl to access it:

/tmp/grpc7-v1.39.0-go-1.21-static &
curl -v http://127.0.0.1:3000/grpc

[damemi]

Thanks for the detailed response and reproducer @bittercookie, I'm working on a fix for this

[damemi]

Opened #1235 to fix this. @bittercookie unfortunately we decided to take the approach of only supporting Status code reporting for >= 1.40 (the current error format at the time we added this feature) rather than add conditionals for every past version of gRPC. However this should allow you to run it without the fatal error