panic: runtime error: invalid memory address or nil pointer dereference

[0michalsokolowski0]

Description

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x78 pc=0xb7e584]

goroutine 46 [running]:
go.opentelemetry.io/otel/sdk/trace.(*forceFlushSpan).Resource(0x1217600?)
:1 +0x24
go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/tracetransform.Spans({0xc0002d6000, 0x2, 0xc0000966c0?})
/go/pkg/mod/go.opentelemetry.io/otel/exporters/otlp/otlptrace@v1.16.0/internal/tracetransform/span.go:47 +0x183
go.opentelemetry.io/otel/exporters/otlp/otlptrace.(*Exporter).ExportSpans(0xc0000a9d10, {0x164ebf0, 0xc003d17650}, {0xc0002d6000?, 0xc0002d9160?, 0xc00081ec00?})
/go/pkg/mod/go.opentelemetry.io/otel/exporters/otlp/otlptrace@v1.16.0/exporter.go:44 +0x3a
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).exportSpans(0xc0002072c0, {0x164eb80, 0xc0039680f0?})
/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:269 +0x264
go.opentelemetry.io/otel/sdk/trace.(*batchSpanProcessor).drainQueue(0xc0002072c0)
/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:330 +0x275
go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor.func1()
/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:129 +0x6f
created by go.opentelemetry.io/otel/sdk/trace.NewBatchSpanProcessor
/go/pkg/mod/go.opentelemetry.io/otel/sdk@v1.16.0/trace/batch_span_processor.go:126 +0x2ea

Environment

• go.opentelemetry.io/otel v1.11.2
• tlptrace v1.16.0

Expected behavior

No panic.

[dmathieu]

It looks like your Tracer Provider has a nil resource.

opentelemetry-go/sdk/trace/span.go

Line 628 in f6a658c

func (s *recordingSpan) Resource() *resource.Resource {

Could you give us a short reproduction example? The way you configure your SDK is probably related to the error here.

[0michalsokolowski0]

Could you give us a short reproduction example? The way you configure your SDK is probably related to the error here.

Thanks for the suggestion, I will check that.

Shouldn't there be nil check here? https://github.com/open-telemetry/opentelemetry-go/blob/main/exporters/otlp/otlptrace/internal/tracetransform/span.go#L47
Considering Resource() is pointer

[dmathieu]

Usually, a provider with a nil resource would be considered an error. There should always be one, even if it's empty.

[pellared]

@0michalsokolowski0 Any update?

[0michalsokolowski0]

	res, err := getResource(ctx)
	if err != nil {}

	exporter, err := createExporter(ctx)
	if err != nil {}

	bsp := trace.NewBatchSpanProcessor(exporter)
        ...
}

func getResource(ctx context.Context) (*resource.Resource, error) {
	hostname, _ := os.Hostname()

	return resource.New(
		ctx,
		resource.WithDetectors(gcp.NewDetector()),
		resource.WithTelemetrySDK(),
		resource.WithContainer(),
		resource.WithFromEnv(),
		resource.WithAttributes(
			semconv.ServiceNameKey.String(config.ServiceName()),
			semconv.ServiceNamespaceKey.String("backend"),
			semconv.ServiceInstanceIDKey.String(hostname),
			semconv.DeploymentEnvironmentKey.String(config.EnvID()),
		),
	)
}

func createExporter(ctx context.Context) (*otlptrace.Exporter, error) {
	ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
	defer cancel()

	conn, err := grpc.DialContext(
		ctx,
		config.String("tracing.collector_addr"),
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithBlock(),
	)
	if err != nil {}
	
	exporter, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
	if err != nil {
		return nil, err
	}

	return exporter, nil
}

[pellared]

Can you please provide a working example?

[0michalsokolowski0]

No, I cannot. It randomly killed two pods out of N and happened at least once.

Usually, a provider with a nil resource would be considered an error.

Triggering this seems like the easiest way to reproduce it to me.

[pellared]

ider with a nil resource would be considered an error.

Triggering this seems like the easiest way to reproduce it to me.

Have you tried it? I do not think it is possible in the latest version.

[0michalsokolowski0]

I am afraid I don't follow.
There is a pointer that shouldn't ever be nil, but surprisingly it is nil.
Afaik it should be fixed first. I don't have time to try to reproduce something happening randomly now and then.
If that's a problem, I can close the issue.

[pellared]

I have finally found the bug. See bb0f44f. Thanks for reporting 👍