Opentelemetry - what is the right configuration?

[vasigorc]

Hello 👋

We are trying to understand what is the expected way to integrate Opentelemetry into our project for the following objective:

1. Gather traces and metrics specific to our application
2. Export them through opentelemetry-collector
3. Ingest and observe them in Elastic-Kibana stack

Before

Previously we were able to successfully export traces only by using javaagent 1.6.0 and the following dependencies:

lazy val otelVersion = "1.6.0"
...
val compileDependencies: Seq[ModuleID] = Seq(
"io.opentelemetry" % "opentelemetry-api" % otelVersion,
"io.opentelemetry" % "opentelemetry-exporter-otlp" % otelVersion,
"io.opentelemetry" % "opentelemetry-extension-annotations" % otelVersion,
"io.opentelemetry" % "opentelemetry-sdk" % otelVersion,
"io.opentelemetry.instrumentation" % "opentelemetry-logback-1.0" % "1.4.1-alpha",

The way we obtained tracer was the following:

def getTracer: Tracer =  GlobalOpenTelemetry.getTracer(serviceName)

Recently

...we set as a goal to switch on metrics through opentelemetry as well. We've updated our dependencies (including the agent) as described here and we set-up our meter using SDKMeterProvider as described here 👈 in the linked comment otel.metrics.exporter is set to none, normally it was set to otlp. With that we were seing some metrics in opentelemetry collector, but that had severe performance impact as discussed in this thread.

Lastly

As the outcome of the discussion from the last link we've removed non opentelemetry-api dependencies and switched to using meter derived from global scope:

def getMeter: OtelMeter = GlobalOpenTelemetry.getMeter(serviceName)

Unfortunately now we cannot even see our metrics in the collector at all. That seems to be in line with what the documentation says:

Unless your application imports the OpenTelemetry SDK, your instrumentation does nothing and does not impact application performance.

Issue

Together with the previous two discussions (linked in this thread) I find this confusing. I would like to understand what dependencies, versions and configurations we need to get to our objective. Do we need to rely on GlobalOpenTelemetry or is it considered bad? Thank you in advance for any help or relevant advice.

The collector yaml file that I use for local investigation is below.

receivers:
  otlp/receiver:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 10s
  memory_limiter:
    check_interval: 1s
    limit_mib: 200
    spike_limit_mib: 100

exporters:
  otlp/elastic:
    endpoint: apm-server:8200
    tls:
      insecure: true
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp/receiver]
      processors: [memory_limiter,batch]
      exporters: [otlp/elastic]
    metrics:
      receivers: [otlp/receiver]
      processors: [memory_limiter,batch]
      exporters: [otlp/elastic, logging]

I use opentelemetry-collector version 0.46.0 and javaagent version 1.12.0.

[jack-berg]

If you're using the agent, it will automatically configure an OpenTelemetry instance using the autoconfigure module. All of the configuration properties used by autoconfigure can be used with the agent. If those options are for some reason insufficient, there are a variety of extension options available.

If your application has additional instrumentation and needs to access the OpenTelemetry instance, it can do so by calling GlobalOpenTelemetry.get(). Its recommended to only call GlobalOpenTelemetry.get() once and use the result to initialize any application components that are dependent on it. Your application should not configure a separate instance of OpenTelemetry and set it globally via GlobalOpenTelemetry.set(..).

In terms of which dependencies you need, since SDK configuration is happening in the agent, you should only need to depend on opentelemetry-api. (That is, in addition to the instrumentation tools your application uses opentelemetry-extension-annotations and opentelemetry-logback-mdc-1.0, which replaces opentelemetry-logback-1.0)

If you're using version 1.12.0 of the agent, you should use version 1.12.0 of all dependencies managed by opentelemetry-java and opentelemetry-java-instrumentation. We publish a variety of boms to assist with this: opentelemetry-bom, opentelemetry-bom-alpha, opentelemetry-instrumentation-bom-alpha

[rvoodoochilep]

Thanks for the answer, Jack!

I have a clarifying question:

You state, "If you're using the agent, it will automatically configure an OpenTelemetry instance using the autoconfigure module". Let's call this autoconfigured OpenTelemetry instance "Instance A".

Next, "If your application has additional instrumentation and needs to access the OpenTelemetry instance, it can do so by calling GlobalOpenTelemetry.get()."

Am I correct in saying:
Given all of the above, and if one is using an agent, writing GlobalOpenTelemetry.get() in application code will return "Instance A", correct?

Thanks in advance! Was curious after finding this helpful thread.

[jack-berg]

Correct!

[rvoodoochilep]

Nice!

[anuraaga]

While metrics has been alpha, we don't enable it by default so you may just be missing OTEL_METRICS_EXPORTER=otlp in the env vars for your app. We're hoping to enable this by default in the next release.

[trask]

hi @vasigorc, all of these steps should be unnecessary, can you post a small repro so we can see where the disconnect is between what we are expecting and what you are seeing?

[vasigorc]

I will try to work something out.

[vasigorc]

Hello @trask I must say it is both embarrassing and hilarious for me 🙂 Here's my intended small repro project and respectively here is the list of the used dependencies - everything works as you expected, below is an example of logs:

otel-collector    | InstrumentationLibraryMetrics #1
otel-collector    | InstrumentationLibraryMetrics SchemaURL: 
otel-collector    | InstrumentationLibrary our-app 
otel-collector    | Metric #0
otel-collector    | Descriptor:
otel-collector    |      -> Name: AppCounter
otel-collector    |      -> Description: An example of counter
otel-collector    |      -> Unit: 1
otel-collector    |      -> DataType: Sum
otel-collector    |      -> IsMonotonic: true
otel-collector    |      -> AggregationTemporality: AGGREGATION_TEMPORALITY_CUMULATIVE
otel-collector    | NumberDataPoints #0
otel-collector    | Data point attributes:
otel-collector    |      -> some-attribute: STRING(feeacf9f-c685-4882-bf15-f5b6558977a4)
otel-collector    | StartTimestamp: 2022-03-30 17:18:09.389933 +0000 UTC
otel-collector    | Timestamp: 2022-03-30 17:19:49.417492 +0000 UTC
otel-collector    | Value: 23
otel-collector    | NumberDataPoints #1

Not sure why same (similar?) setup wouldn't work for our enterprise app I will look closer into that. Please accept my apologies. I will remark this as answered.

[jack-berg]

Software is hard! 😁

[trask]

no worries! good luck finding the issue in the real app, and don't hesitate to post back if you are able to narrow it down to some other reproducible case that we can take a look at