[exporter/datadog] collector panics on invalid trace & span ids in logs

[kevinnoel-be]

Component(s)

exporter/datadog

What happened?

Description

We have one (internal) service that is producing logs with invalid sized trace & span IDs (32 & 19 bytes length respectively). When our OpenTelemetry collector processes those specific logs the collector panics and gets restarted until next problematic log line (Kubernetes deployment). Any other data (logs, metrics & traces) are getting dropped/missed.

Steps to Reproduce

• configure a collector with the datadog exporter
• collect logs with invalid sized trace/span IDs

Expected Result

When trace_id and/or span_id parsing fails on invalid size (or other), the collector doesn't crash and those attributes are forwarded as they are (or are ignored).

Actual Result

The datadog exporter panics on those log lines and we're losing data.

Collector version

v0.100.0

Environment information

Environment

OS: Ubuntu & scratch based Docker image
Compiler: go 1.21.11

OpenTelemetry Collector configuration

receivers:
  filelog:
    include:
      - invalid-trace-span-ids.jsonl
    start_at: beginning
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.timestamp
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
        severity:
          parse_from: attributes.status

processors:

exporters:
  debug:
    verbosity: detailed

  datadog:
    api:
      key: "${file:.dd-api-key}"

service:
  pipelines:
    logs:
      receivers:
        - filelog
      processors:
      exporters:
        - debug
        - datadog

Log output

panic: runtime error: index out of range [8] with length 8

goroutine 97 [running]:
encoding/hex.Decode({0xc04b042fd8?, 0xc04d628b28?, 0x6622456?}, {0xc04b043030?, 0x1?, 0xc04b042ff0?})
	encoding/hex/hex.go:91 +0x130
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs.decodeSpanID(...)
	github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs@v0.16.0/transform.go:233
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs.transform.func1({0xc04d92ebb8, 0x7}, {0xc04e4c7970?, 0xc04d614af4?})
	github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs@v0.16.0/transform.go:104 +0x4a5
go.opentelemetry.io/collector/pdata/pcommon.Map.Range({0xc001f41378?, 0xc04d614af4?}, 0xc04b043478)
	go.opentelemetry.io/collector/pdata@v1.9.0/pcommon/map.go:222 +0x97
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs.transform({0xc001f41340?, 0xc04d614af4?}, {0xc0496ad2ef, 0xe}, {0xc001984cff, 0xc}, {0xc04d800a98?, 0xc04d614af4?}, 0xc0019b0100)
	github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs@v0.16.0/transform.go:84 +0x229
github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs.(*Translator).MapLogs(0xc0000d18c0, {0x71d0058, 0xc04ebd20f0}, {0xc04d76e5b8?, 0xc04d614af4?})
	github.com/DataDog/opentelemetry-mapping-go/pkg/otlp/logs@v0.16.0/translator.go:93 +0x365
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter.(*logsExporter).consumeLogs(0xc0001d04b0, {0x71d0058?, 0xc04ebd20f0?}, {0xc04d76e5b8?, 0xc04d614af4?})
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter@v0.100.0/logs_exporter.go:120 +0x14a
go.opentelemetry.io/collector/exporter/exporterhelper.(*logsRequest).Export(0xc04dcf9c80?, {0x71d0058?, 0xc04ebd20f0?})
	go.opentelemetry.io/collector/exporter@v0.100.0/exporterhelper/logs.go:59 +0x31

Additional context

An example of log with invalid trace/span IDs:

{ "timestamp": "2024-06-14T10:50:01.860Z", "status": "info", "message": "some log with invalid trace & span IDs", "trace_id": "d8f47f62378c2301657f395a30d6533c", "span_id": "2023675201651514964"}

The problem lies on the lack of size check here & here

[github-actions]

Pinging code owners:

• exporter/datadog: @mx-psi @dineshg13 @liustanley @songy23 @mackjmr @ankitpatel96

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[songy23]

Thanks for reporting & looking into this @kevinnoel-be looks like you have a fix proposed in DataDog/opentelemetry-mapping-go#340, we will take a look

[kevinnoel-be]

I see the fix has been merged, but we still need to have a released version there + updating in the exporter, no?

[mx-psi]

Yes, this was autoclosed but let me reopen it until we handle that :)

[kevinnoel-be]

Thanks! :)