Excessive number of open file descriptors for the app when using otlp exporter

[vasigorc]

The issue
We have an app that exports traces and metrics. Here is how our otel.properties configuration file look like:

otel.service.name=our-app
otel.service.language.name=scala
otel.resource.attributes=service.namespace=APP
otel.metrics.exporter=otlp

Whenever we deploy the app it starts exploding with open file descriptors for the application process. Here is the maximum number of file handles for our app:

[ouruser@ip-local-ip ~]$ grep "Max open files" /proc/$pid/limits
Max open files            4096                 4096                 files

But as we start the app the number starts slowly growing up. That seems to be linear to receiving new traces/metrics, but we can't prove this:

[ouruser@ip-local-ip ~]$ sudo lsof -p $pid | grep "localhost:4317" | wc -l
4299

A typical line in the list of files looks like this:

java    16419 ouruser  209u  IPv4           15097110       0t0      TCP localhost:56922->localhost:4317 (ESTABLISHED)

We have opentelemetry-collector running at localhost:4317.

Versions

Libraries

// we use sbt
  lazy val otelVersion = "1.12.0"
  lazy val otlpMetricsVersion = "1.11.0-alpha"
...
 val compileDependencies: Seq[ModuleID] = Seq(
    "io.opentelemetry" % "opentelemetry-api" % otelVersion,
    "io.opentelemetry" % "opentelemetry-exporter-otlp" % otelVersion,
    "io.opentelemetry" % "opentelemetry-exporter-otlp-metrics" % otlpMetricsVersion,
    "io.opentelemetry" % "opentelemetry-extension-annotations" % otelVersion,
    "io.opentelemetry" % "opentelemetry-sdk" % otelVersion,
    "io.opentelemetry" % "opentelemetry-sdk-metrics" % otlpMetricsVersion,
    "io.opentelemetry.instrumentation" % "opentelemetry-logback-mdc-1.0" % otlpMetricsVersion,

Agent

-javaagent:src/pack/lib/opentelemetry/opentelemetry-javaagent-all-v1.11.1.jar

Otel Collector (through 🐳 ):

  otel-collector:
    container_name: otel-collector
    image: otel/opentelemetry-collector-contrib:0.46.0
    command: [ "--config=/etc/otel-collector-config.yml" ]
    working_dir: "/project"
    volumes:
      - ${PWD}/:/project
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
      - ./company/opentelemetry-jmx-metrics.jar:/opentelemetry-jmx-metrics.jar
    ports:
      - "4317:4317"
      - "4318:4318"
    depends_on:
      - apm-server
    networks:
      - elastic

Environment
Compiler: OpenJDK 8 (built on Jenkins server)
Runtime OS: 4.14.268-139.500.amzn1.x86_64
Runtime JDK: OpenJDK Runtime Environment (build 1.8.0_312-b07)

[Updated] Also attaching a sanitized copy of our otel-collector-config.yml file:

receivers:
  otlp/receiver:
    protocols:
      grpc:
      http:

exporters:
  otlp/elastic:
    endpoint: <REDACTED>
    headers:
      Authorization: "Bearer <REDACTED>"
  logging:

processors:
  batch:
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_mib: 200
    spike_limit_mib: 100
  resource:
    attributes:
    - key: deployment.environment
      value: <REDACTED>
      action: upsert
    - key: service.environment
      value: <REDACTED>
      action: upsert

service:
  pipelines:
    traces:
      receivers: [otlp/receiver]
      processors: [memory_limiter,batch,resource]
      exporters: [otlp/elastic]
    metrics:
      receivers: [otlp/receiver]
      processors: [memory_limiter,batch,resource]
      exporters: [otlp/elastic]

[anuraaga]

Hi @vasigorc - are you initializing the SDK in the code and also using the javaagent? This is not correct usage - we'd expect your list of dependencies to only include opentelemetry-api and nothing else. If you are initializing the SDK, maybe we can start by removing that and see if the issue still remains?

[vasigorc]

Hello @anuraaga - thank you for following up! Yes I thought that too based on Jack's response here.

list of dependencies to only include opentelemetry-api

1. We can't remove opentelemetry-extension-annotations as we use these for @WithSpan annotations for tracing
2. Also opentelemetry-logback-mdc-1.0 is being used for logging
3. opentelemetry-exporter-otlp and opentelemetry-exporter-otlp-metrics aren't marked as SDKs - should we get rid of them too?

If you are initializing the SDK, maybe we can start by removing that

I wanted to confirm whether you meant to remove just dependencies or initialization as well. If we remove the initialization then we'd only be able to use GlobalOpenTelemetry to get or construct MeterProvider | Meter?

[anuraaga]

If we remove the initialization then we'd only be able to use GlobalOpenTelemetry to get or construct MeterProvider | Meter?

Yes this is the way to use OTel when using the agent. I think an improvement for the agent to automatically inject the OpenTelemetry into DI like spring or dagger would help with this but we haven't implemented that so far. We generally recommend using GlobalOpenTelemetry exactly once in an app and pass the result where needed if possible.

[anuraaga]

Filed open-telemetry/opentelemetry-java-instrumentation#5682 to see if we can make this more obvious. If you see any obvious docs to improve a PR or pointer will be very helpful :)

[vasigorc]

I've found some traces which could have impacted our decision. Looking at the official documentation we read:

If your artifact is a standalone process or service, then you would take a dependency on the API and the SDK.

and

If you’re building a service process, you’ll also need to configure the SDK with appropriate options for exporting your telemetry data to some analysis backend. We recommend that this configuration be handled programmatically through a configuration file or some other mechanism.

☝️ this is our use case. We have a Elastic-Kibana backend that we are pushing data through opentelemetry collector.

After removing all sdk dependencies, and SDK initialization logic I no longer see any app specific traces/measurements on the collector side. I will start a new discussion for that and mark this as done.