Nested env variables not resolved, unable to use downward API

[tiithansen]

Component(s)

auto-instrumentation

What happened?

Description

We want traces to be exporter to OTEL collector running on the same host/node. For that purpose we use downward API to get status.hostIP and reference it in OTEL_EXPORTER_OTLP_ENDPOINT end, but OTEL operator does not resolve it and exporter gets invalid IP.

Steps to Reproduce

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: instrumentation
spec:
  propagators:
  - tracecontext
  - baggage
  sampler:
    type: parentbased_traceidratio
    argument: {{ .Values.instrumentation.sampler.parentbased_traceidratio | quote }}
  env:
  - name: OTEL_BSP_MAX_EXPORT_BATCH_SIZE
    value: "512"
  - name: OTEL_EXPORTER_HOST_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  nodejs:
    env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "http://$(OTEL_EXPORTER_HOST_IP):4317"

Expected Result

Env OTEL_EXPORTER_OTLP_ENDPOINT should contain correct address with host ip.

Actual Result

Env OTEL_EXPORTER_OTLP_ENDPOINT contains string as http://$(OTEL_EXPORTER_HOST_IP):4317 instead which is not resolved.

Kubernetes Version

1.28

Operator version

version: 0.47.0 appVersion: 0.93.0

Collector version

Custom

Environment information

No response

Log output

No response

Additional context

No response

[pavolloffay]

instrumentation: Instrumentation now automatically add node and pod ips as env vars OTEL_NODE_IP and OTEL_POD_IP to instrumented containers. (#2769)

The env vars with the IPs are already injected

[pavolloffay]

Could you please paste here the pod spec after the instrumentation is injected?

[pavolloffay]

This is the code for setting the exporter endpoint

opentelemetry-operator/pkg/instrumentation/sdk.go

Line 288 in 5bafbfd

if otelinst.Spec.Exporter.Endpoint != "" {

I am not sure why it should not work

[fyuan1316]

Expected Result
Env OTEL_EXPORTER_OTLP_ENDPOINT should contain correct address with host ip.

opentelemetry-operator/pkg/instrumentation/sdk.go

Lines 73 to 76 in 5bafbfd

	pod = i.injectCommonEnvVar(otelinst, pod, index)
	pod = i.injectCommonSDKConfig(ctx, otelinst, ns, pod, index, index)
	pod = i.setInitContainerSecurityContext(pod, pod.Spec.Containers[index].SecurityContext, javaInitContainerName)
	}

After looking at the source code implementation, I think there might be some misunderstanding here.

By design both nodejs.env and Instrumentation spec.env end up injecting pod container[0].env. No string substitution occurs during the injection process. Therefore, the value of OTEL_EXPORTER_OTLP_ENDPOINT will always be "http://$(OTEL_EXPORTER_HOST_IP):4317".

So this could be a potential requirement rather than a bug? @pavolloffay @tiithansen