go.opentelemetry.io/otel/sdk/metric: meter provider created with WithResource does not honor resource-related environment variables (OTEL_RESOURCE_ATTRIBUTES)

[basti1302]

Description

Consider a meter provider created like this:

theResource, err := resource.New(ctx, resource.WithAttributes(semconv.K8SNodeName("my-k8s-node-name")))

...

meterProvider := metric.NewMeterProvider(
    metric.WithResource(theResource),
    metric.WithReader(metric.NewPeriodicReader(metricExporter,
        metric.WithInterval(3*time.Second))),
)

Also, assume the process has been started with resource attributes being provided via environment variables, e.g. OTEL_RESOURCE_ATTRIBUTES.

The metrics produced by this meter provider will only have the resource attributes from the in-code configuration, the key-value pairs from the environment variable will not be used.

If I read the spec correctly, the env vars should be merged with values provided via in-code config. This is also what happens when creating a tracer provider in a similar fashion. At the very least, the behavior is inconsistent between meter provider and tracer provider (I did not test logger provider yet).

Links to sources:

• opentelemetry-go/sdk/trace/provider.go

  Line 353 in 932a4d8

  cfg.resource, err = resource.Merge(resource.Environment(), r)

  -- trace provider calls resource.Merge(resource.Environment(), r)

• opentelemetry-go/sdk/metric/config.go

  Line 106 in 932a4d8

  conf.res = res

  -- meter provider does not seem to merge with resource.Environment()

I've created an SSCCE/reproducer here: https://github.com/basti1302/opentelemetry-go-metric-resource-reproducer. Its README lists the produced metrics and spans. Here is an excerpt:

metric:

{
  "Resource": [
    {
      "Key": "k8s.node.name", // <- Wrong: Only the attributes from the in-code config are here.
      "Value": {
        "Type": "STRING",
        "Value": "my-k8s-node-name"
      }
    }
  ],
  "ScopeMetrics": [...]
}

span: 

{
	"Name": "/",
        ...
	"Resource": [             // <- Correct: The attributes from the environment variable are used here as well.
		{
			"Key": "k8s.node.name",
			"Value": {
				"Type": "STRING",
				"Value": "my-k8s-node-name"
			}
		},
		{
			"Key": "service.name",
			"Value": {
				"Type": "STRING",
				"Value": "my-service-name"
			}
		},
		{
			"Key": "service.namespace",
			"Value": {
				"Type": "STRING",
				"Value": "my-service-namespace"
			}
		},
		{
			"Key": "service.version",
			"Value": {
				"Type": "STRING",
				"Value": "1.2.3"
			}
		}
	],
...
}

Environment

• OS: Linux, MacOS
• Architecture: x86_64, arm
• Go Version: 1.22
• opentelemetry-go version: v1.29.0

Steps To Reproduce

See https://github.com/basti1302/opentelemetry-go-metric-resource-reproducer/blob/main/README.md

Expected behavior

The meter provider should merge key-value pairs from the environment with the in-code configuration.

[basti1302]

For the record, I would also be interested in contributing a fix. But before working on that I would like a second opinion on the desired behavior - e.g. is the merging happening in trace provider correct (I thin so) or is meter provider's behavior correct.

[dmathieu]

I agree that merging both resources should be the proper behavior.

[basti1302]

Created #5773 which will fix this for metrics as well as for logs, where it was equally broken.