[opentelemetry-instrumentation-aws-lambda] cloud.account.id resource detection gets mapped to labels.cloud_account_id in elasticsearch

[calindurnea]

What version of OpenTelemetry are you using?

"@opentelemetry/api": "1.9.0",
"@opentelemetry/auto-configuration-propagators": "0.3.0",
"@opentelemetry/exporter-metrics-otlp-proto": "0.53.0",
"@opentelemetry/exporter-trace-otlp-proto": "0.53.0",
"@opentelemetry/host-metrics": "0.35.3",
"@opentelemetry/instrumentation": "0.53.0",
"@opentelemetry/instrumentation-aws-lambda": "0.44.0",
"@opentelemetry/instrumentation-aws-sdk": "0.44.0",
"@opentelemetry/instrumentation-http": "0.53.0",
"@opentelemetry/resource-detector-aws": "1.6.1",
"@opentelemetry/resources": "1.26.0",
"@opentelemetry/sdk-metrics": "1.26.0",
"@opentelemetry/sdk-trace-base": "1.26.0",
"@opentelemetry/sdk-trace-node": "1.26.0",
"@opentelemetry/semantic-conventions": "1.27.0"

What version of Node are you using?

20

What did you do?

Using the configuration provided below as a lambda layer, I am instrumenting a lambda.

What did you expect to see?

The cloud.account.id to be part of the Cloud section

What did you see instead?

The cloud.account.id is part of the labels.

I am not sure if this is related to the open-telemetry packages or elasticsearch, but I would like to make sure that there is nothing wrong with this.

Additional context

Instrumentation layer used:

/* eslint-disable no-console */
import {
  context as otelContext,
  diag,
  DiagConsoleLogger,
  DiagLogLevel,
  metrics,
  propagation,
} from "@opentelemetry/api";
import { BatchSpanProcessor } from "@opentelemetry/sdk-trace-base";
import { OTLPTraceExporter } from "@opentelemetry/exporter-trace-otlp-proto";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import {
  detectResourcesSync,
  envDetectorSync,
  hostDetectorSync,
  processDetectorSync,
} from "@opentelemetry/resources";
import { awsLambdaDetectorSync } from "@opentelemetry/resource-detector-aws";
import {
  AggregationTemporality,
  MeterProvider,
  MeterProviderOptions,
  PeriodicExportingMetricReader,
} from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-proto";
import { getPropagator } from "@opentelemetry/auto-configuration-propagators";
import { AwsLambdaInstrumentation } from "@opentelemetry/instrumentation-aws-lambda";
import { AwsInstrumentation } from "@opentelemetry/instrumentation-aws-sdk";
import { HostMetrics } from "@opentelemetry/host-metrics";
import { HttpInstrumentation } from "@opentelemetry/instrumentation-http";

// configure wrapper logging
diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ALL);

type SQSEvent = {
  Records: Array<{
    body: string;
  }>;
};

const traceParentExtractor = (event: SQSEvent) => {
  const traceparents: string[] = [];
  if (!event?.Records || !Array.isArray(event.Records)) {
    console.info(
      "Trace Parent Extractor could not find array under event.Records",
    );
    return traceparents;
  }
  for (const record of event.Records) {
    try {
      traceparents.push(JSON.parse(record.body).detail.traceparent);
    } catch (e) {
      // ignore events that don't have a valid traceparent
    }
  }

  return traceparents;
};

const elasticApmUrl = "the url";

const instrumentations = [
  new AwsLambdaInstrumentation({
    disableAwsContextPropagation: true,
    eventContextExtractor: (event) => {
      const traceparent = traceParentExtractor(event);
      const eventContext = propagation.extract(otelContext.active(), {
        traceparent: traceparent[0],
      });
      return eventContext;
    },
  }),
  new AwsInstrumentation({
    suppressInternalInstrumentation: true,
  }),
  new HttpInstrumentation({
    enabled: true,
  }),
];

// Register instrumentations synchronously to ensure code is patched even before provider is ready.
registerInstrumentations({
  instrumentations,
});

function initializeProvider() {
  const resource = detectResourcesSync({
    detectors: [
      envDetectorSync,
      hostDetectorSync,
      processDetectorSync,
      awsLambdaDetectorSync,
    ],
  });

  const tracerProvider = new NodeTracerProvider({
    resource,
  });

  const spanProcessor = new BatchSpanProcessor(
    new OTLPTraceExporter({
      url: `${elasticApmUrl}/v1/traces`,
      headers: {
        Authorization: `Bearer secret token`,
      },
    }),
  );

  tracerProvider.addSpanProcessor(spanProcessor);
  tracerProvider.register({ propagator: getPropagator() });

  const metricExporter = new OTLPMetricExporter({
    url: `${elasticApmUrl}/v1/metrics`,
    headers: {
      Authorization: `Bearer secret token`,
    },
    temporalityPreference: AggregationTemporality.DELTA,
  });
  const meterConfig: MeterProviderOptions = {
    resource,
    readers: [
      new PeriodicExportingMetricReader({
        exporter: metricExporter,
      }),
    ],
  };

  const meterProvider = new MeterProvider(meterConfig);
  metrics.setGlobalMeterProvider(meterProvider);

  const hostMetrics = new HostMetrics({ meterProvider });
  hostMetrics.start();

  // Re-register instrumentation with initialized provider. Patched code will see the update.
  registerInstrumentations({
    instrumentations,
    tracerProvider,
    meterProvider,
  });
}

initializeProvider();

[trentm]

Hi @calindurnea. I work at Elastic on Observability things, though not on the components responsible for ingesting data into Elasticsearch.

This behaviour, mapping resource attributes to labels.*, is currently how APM server (the current component handling ingest) works, so this isn't a bug -- and certainly not a bug with OpenTelemetry. There was a thread covering this on the #otel-collector channel of the CNCF slack a little while back: https://cloud-native.slack.com/archives/C01N6P7KR6W/p1718970020370909?thread_ts=1718955152.426309&cid=C01N6P7KR6W

(If you aren't able to read that slack thread, please let me know and I can reproduce the relevant comments here.)

This mapping isn't ideal, and Elastic has plans to improve the ingest of OTLP data to be more native, but that is the way it works currently. That thread explains the reasons why better, in general. Why the cloud.account.id resource attribute is not yet mapped to Elastic's ECS cloud.account.id field yet, I don't know.

[axw]

@trentm in this case, the bug is with the instrumentation. cloud.account.id is a resource attribute, but the instrumentation is setting it as a span attribute:

opentelemetry-js-contrib/plugins/node/opentelemetry-instrumentation-aws-lambda/src/instrumentation.ts

Lines 239 to 254 in 9b43ccb

	const span = plugin.tracer.startSpan(
	name,
	{
	kind: SpanKind.SERVER,
	attributes: {
	[SEMATTRS_FAAS_EXECUTION]: context.awsRequestId,
	[SEMRESATTRS_FAAS_ID]: context.invokedFunctionArn,
	[SEMRESATTRS_CLOUD_ACCOUNT_ID]:
	AwsLambdaInstrumentation._extractAccountId(
	context.invokedFunctionArn
	),
	[ATTR_FAAS_COLDSTART]: requestIsColdStart,
	},
	},
	parent
	);

(APM Server does understand cloud.account.id specifically, but only as a resource attribute.)

[calindurnea]

Hello @axw,
thank you very much for checking on this and answering.

From your understanding is this something that has a low level of complexity to fix? if that would be the case, I might give it a try :)

[axw]

@calindurnea I'm not sure to be honest, I'm not familiar enough with opentelemetry-js or the Lambda instrumentation. Hopefully @trentm will know 🤞

[calindurnea]

hello @trentm, is this something you can share some knowledge on?