feat: replace OtlpPipeline with exporter builders #2221

pitoniak32 · 2024-10-19T01:35:46Z

Migration Guide

To move from opentelemetry_otlp::new_exporter(), and opentelemetry_otlp::new_pipeline().{tracing,metrics,logging}(), you will need to select the Exporter, and Provider for the signal you are using.

Below there is a guide with details about Exporters and Providers. There are also Examples for each type of signal.

Exporter Guide

The following are the available exporters:

SpanExporter
MetricsExporter
LogExporter

The exporter interface should have the same options as with the old method. For example, a tonic grpc exporter being configured old vs new:

// old
let old_exporter = opentelemetry_otlp::new_exporter()
  .tonic()
  .with_endpoint("http://localhost:4317");

// new
let new_exporter = SpanExporter::builder()
  .tonic()
  .with_endpoint("http://localhost:4317")
  .build()?; // note: you need to build the exporter after setting configuration.

Provider Guide

The following are the available providers:

TracingProvider
SdkMeterProvider
LoggerProvider

The provider interface should be similar to the old method. For example, a TracerProvider being configured old vs new:

// old
let tracer_provider: TracerProvider = opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(old_exporter /* ^^ See exporter guide above ^^ */)
    .with_trace_config(Config::default().with_resource(RESOURCE.clone()))
    .install_batch(runtime::Tokio)?;

// new
let tracer_provider: TracerProvider = TracerProvider::builder()
    .with_batch_exporter(new_exporter /* ^^ See exporter guide above ^^ */, runtime::Tokio)
    .with_config(Config::default().with_resource(RESOURCE.clone()))
    .build(); // note: you need to build the provider after setting configuration.

Signal Examples

Trace Example

// old
fn init_tracer_provider() -> Result<sdktrace::TracerProvider, TraceError> {
    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint("http://localhost:4317"),
        )
        .with_trace_config(Config::default().with_resource(RESOURCE.clone()))
        .install_batch(runtime::Tokio)
}

// new
fn init_tracer_provider() -> Result<sdktrace::TracerProvider, TraceError> {
    let exporter = SpanExporter::builder()
        .with_tonic()
        .with_endpoint("http://localhost:4317")
        .build()?;
    Ok(sdktrace::TracerProvider::builder()
        .with_config(Config::default().with_resource(RESOURCE.clone()))
        .with_batch_exporter(exporter, runtime::Tokio)
        .build())
}

Metric Example

// old
fn init_metrics() -> Result<opentelemetry_sdk::metrics::SdkMeterProvider, MetricsError> {
    let export_config = ExportConfig {
        endpoint: "http://localhost:4317".to_string(),
        ..ExportConfig::default()
    };
    opentelemetry_otlp::new_pipeline()
        .metrics(runtime::Tokio)
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_export_config(export_config),
        )
}

// new
fn init_metrics() -> Result<opentelemetry_sdk::metrics::SdkMeterProvider, MetricsError> {
    let exporter = MetricsExporter::builder().with_tonic().build()?;
    let reader = PeriodicReader::builder(exporter, runtime::Tokio).build();

    Ok(SdkMeterProvider::builder()
        .with_reader(reader)
        .with_resource(RESOURCE.clone())
        .build())
}

Log Example

// old
fn init_logs() -> Result<opentelemetry_sdk::logs::LoggerProvider, LogError> {
    opentelemetry_otlp::new_pipeline()
        .logging()
        .with_resource(RESOURCE.clone())
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint("http://localhost:4317"),
        )
        .install_batch(runtime::Tokio)
        .with_batch_exporter(exporter, runtime::Tokio)
        .build())
}

// new
fn init_logs() -> Result<opentelemetry_sdk::logs::LoggerProvider, LogError> {
    let exporter = LogExporter::builder()
        .with_tonic()
        .with_endpoint("http://localhost:4317")
        .build()?;
  
    Ok(LoggerProvider::builder()
        .with_resource(RESOURCE.clone())
        .with_batch_exporter(exporter, runtime::Tokio)
        .build())
}

Changes

Started to replace the OTLPPipeline pattern in the opentelemetry-otlp crate.

Merge requirement checklist

CONTRIBUTING guidelines followed
Unit tests added/updated (if applicable)
Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
Changes in public API reviewed (if applicable)

pitoniak32 · 2024-10-19T01:37:42Z

I have started looking into replicating opentelemetry-stdout patterns. I have gotten to a place that with very basic and defaulted behavior its similar. I just wanted to confirm that it seems reasonable before I start getting into the details of allowing more customization to the exporter builders.

If you have a chance to review this @cijothomas I would really appreciate it!

pitoniak32 · 2024-10-19T13:55:42Z

This might be another issue / PR, but I'm noticing the SdkMeterProvider and TracerProvider have different interfaces for configuring options like Resource for example. I have a few ideas for possibly unifying this if its open for discussion in another location?

codecov · 2024-10-19T18:53:46Z

Codecov Report

Attention: Patch coverage is 62.04380% with 104 lines in your changes missing coverage. Please review.

Project coverage is 79.6%. Comparing base (a7de955) to head (d6ab3de).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
opentelemetry-otlp/src/metric.rs	0.0%	33 Missing ⚠️
opentelemetry-otlp/src/logs.rs	0.0%	31 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/mod.rs	78.7%	23 Missing ⚠️
opentelemetry-otlp/src/span.rs	53.8%	12 Missing ⚠️
opentelemetry-otlp/src/exporter/http/mod.rs	92.7%	5 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2221     +/-   ##
=======================================
+ Coverage   79.1%   79.6%   +0.4%     
=======================================
  Files        121     121             
  Lines      21171   21094     -77     
=======================================
+ Hits       16762   16805     +43     
+ Misses      4409    4289    -120

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

cijothomas · 2024-10-20T04:34:18Z

This might be another issue / PR, but I'm noticing the SdkMeterProvider and TracerProvider have different interfaces for configuring options like Resource for example. I have a few ideas for possibly unifying this if its open for discussion in another location?

Yes, ensuring consistency across signals wherever feasible is definitely something we need to tackle soon!

examples/self-diagnostics/src/main.rs

opentelemetry-sdk/src/metrics/exporter.rs

opentelemetry-stdout/examples/basic.rs

opentelemetry-otlp/src/exporter/mod.rs

cijothomas · 2024-10-20T05:33:04Z

I have started looking into replicating opentelemetry-stdout patterns. I have gotten to a place that with very basic and defaulted behavior its similar. I just wanted to confirm that it seems reasonable before I start getting into the details of allowing more customization to the exporter builders.

If you have a chance to review this @cijothomas I would really appreciate it!

Took a brief look and I like this direction. Left some comments, mostly about limiting the scope of PR. We really prefer short PRs each solving one particular aspect.

pitoniak32 · 2024-10-20T12:23:31Z

Took a brief look and I like this direction. Left some comments, mostly about limiting the scope of PR. We really prefer short PRs each solving one particular aspect.

I'll go through and remove some of the changes I made that weren't specific to this issue 👍

done

pitoniak32 · 2024-10-20T13:41:14Z

As an output of this PR I would like to open a discussion / issue to talk about the following refactors:

Renaming the types named Metics* to Metric* to follow the spec.
Renaming opentelemetry::metrics::Result to opentelemetry::metrics:MetricResult (or doing the opposite and using the mod path for LogResult, and TraceResult).
Modifying the TracerProvider to follow a similar configuration approach that is used for the other "Providers" instead of or in addition to the with_config(...) option for consistency across signals.

these modules are conditionally included by the same flags in lib

examples/self-diagnostics/src/main.rs

cijothomas · 2024-10-21T03:45:20Z

opentelemetry-otlp/examples/basic-otlp-http/src/main.rs

+fn init_logs() -> Result<sdklogs::LoggerProvider, opentelemetry::logs::LogError> {
+    let exporter_builder = LogExporter::builder()
+        .with_http()
+        .with_endpoint("http://localhost:4318/v1/logs")


is it possible for endpoint, protocol to take defaults, so users does not have to specify this (unless they want nondefaults)

good call out, The defaults should already be getting used as they were before, including the http appending the appropriate path. I added a couple more test cases for tonic to cover this.

I also changed the type of the endpoint field to Option<String> so its more explicit that it could be empty, and its on the Builders to handle resolving it.

cijothomas · 2024-10-21T03:56:19Z

opentelemetry-otlp/examples/basic-otlp/src/main.rs

-        .logging()
+    let exporter = LogExporter::builder()
+        .with_tonic()
+        .with_endpoint("http://localhost:4317")


one inconsistency I notice between grpc and http is the end point - one has to append /v1/logs in http, but grpc does not require that. This is something we can address in a follow up.
I am thinking something like:
with_tonic() or with_http() is enough for builder to know which endpoint, protocol to use by default.
builder.with_tonic().build() -- this should just work with defaults
builder.with_http().build() -- this should also just work with defaults.

if user wish to change defaults, then they can use methods like with_endpoint, with_protocol etc.

Lets come back to this after merging this.

sounds good, after my most recent update, the with_{tonic,http}() is enough to determine the http://localhost:431{7,8} and the tests for this are updated also.

I will however need to look at the spec to see how the /v1/* is handled for the two, and I would be happy to create a new pr to get that sorted out.

sounds good, after my most recent update, the with_{tonic,http}() is enough to determine the http://localhost:431{7,8} and the tests for this are updated also.

I will however need to look at the spec to see how the /v1/* is handled for the two, and I would be happy to create a new pr to get that sorted out.

EDIT: After looking back through the code, this should also already be the case, I made some changes to make it a little more clear

opentelemetry-otlp/src/exporter/http/mod.rs

utpilla

Looks good overall from the user-facing changes.

keltia · 2024-11-13T09:41:44Z

Guys, I know this is still 0.x and stuff, but breaking the API on almost every release is, from a user point of view, quite painful. Staying at 0.26 for the moment.

frederikhors · 2024-11-14T20:00:51Z

@pitoniak32, can you help me with with_batch_config(), how to use it now with 0.27?

pitoniak32 · 2024-11-14T21:15:35Z

@pitoniak32, can you help me with with_batch_config(), how to use it now with 0.27?

Sure! Do you have a specific issue? Did you checkout the migration guide in the overview comment already?

frederikhors · 2024-11-14T21:16:29Z

Yeah, but I cannot find with_batch_config() in there.

lalitb · 2024-11-14T21:48:57Z

@frederikhors Can you try something like (not tested)

fn init_tracer_provider() -> Result<sdktrace::TracerProvider, TraceError> {
    // Create a span exporter
    let exporter = SpanExporter::builder()
        .with_http()
        .with_protocol(Protocol::HttpBinary) 
        .with_endpoint("http://localhost:4318/v1/traces")
        .build()?;

    let batch_config = BatchConfigBuilder::default()
        .with_max_queue_size(2048)
        .build();

    let batch_processor = sdktrace::BatchSpanProcessor::builder(exporter, runtime::Tokio)
        .with_batch_config(batch_config)
        .build();

    Ok(TracerProvider::builder()
        .with_span_processor(batch_processor)
        .with_config(Config::default().with_resource(RESOURCE.clone()))
        .build())
}

I think we should modify the basic-otlp-http example to show this more clearly now.

frederikhors · 2024-11-14T22:01:33Z

It works.

I'm having only this error now:

error[E0599]: no method named `tracer` found for struct `opentelemetry_sdk::trace::TracerProvider` in the current scope
   |
87 |             .with_tracer(otlp_tracer_provider.tracer("otlp-tracer"))
   |                                               ^^^^^^
   |
  ::: C:\Users\Fred\.cargo\registry\src\index.crates.io-6f17d22bba15001f\opentelemetry-0.27.0\src\trace\tracer_provider.rs:31:8
   |
31 |     fn tracer(&self, name: impl Into<Cow<'static, str>>) -> Self::Tracer {
   |        ------ the method is available for `opentelemetry_sdk::trace::TracerProvider` here
   |
   = help: items from traits can only be used if the trait is in scope
help: trait `TracerProvider` which provides `tracer` is implemented but not in scope; perhaps you want to import it
   |
1  + use opentelemetry::trace::TracerProvider;
   |
help: there is a method `boxed_tracer` with a similar name
   |
87 |             .with_tracer(otlp_tracer_provider.boxed_tracer("otlp-tracer"))
   |                                               ~~~~~~~~~~~~

for the code:

let layer = tracing_opentelemetry::layer()
    .with_tracer(otlp_tracer_provider.tracer("otlp-tracer"))
    .with_filter(EnvFilter::from("INFO"))
    .boxed();

Importing TracerProvider oviously doesn't work because this is not the fix.

lalitb · 2024-11-14T23:41:01Z

Importing TracerProvider oviously doesn't work because this is not the fix.

Try importing use opentelemetry::trace::TracerProvider as _; if that works?

frederikhors · 2024-11-15T00:30:43Z

Try importing use opentelemetry::trace::TracerProvider as _; if that works?

It works. But ??? Is this OK?

lalitb · 2024-11-15T00:43:45Z

It works. But ??? Is this OK?

Yes, this is typical of Rust — the trait must be in scope for its methods to be accessible.

cijothomas · 2024-11-15T21:49:11Z

Thanks all. If there are any issues/concerns with this change that need help - please open a new issue.

> I followed the migration guide described in open-telemetry/opentelemetry-rust#2221

open-telemetry/opentelemetry-rust#2221 Need a trait to access this now So import it

* fix(deps): update opentelemetry-rust monorepo to 0.27.0 * fix(deps): update rust crate tracing-opentelemetry to 0.28.0 * fix: otel 0.27 tls_config setting open-telemetry/opentelemetry-rust#2221 Need a trait to access this now So import it --------- Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com> Co-authored-by: Philip Cristiano <git@philipcristiano.com>

ref: open-telemetry/opentelemetry-rust#1000 ref: open-telemetry/opentelemetry-rust#2221 ref: open-telemetry/opentelemetry-rust#2085

* chore(deps): cargo * wip: use axum & cargo fmt * wip: remove `once_cell` * wip: upgrade opentelemetry ref: open-telemetry/opentelemetry-rust#1000 ref: open-telemetry/opentelemetry-rust#2221 ref: open-telemetry/opentelemetry-rust#2085 * chore: cargo clippy * chore(deps): upgrade go * version: upgrade to 0.6.0 * chore(deps): use rkyv * chore(deps): update * chore(deps): update * wip: split mods into crates * wip: remove unused dependencies * chore(ci): upgrade rust version in Dockerfile * fix(ci): update cargo install path * chore: tidy up main.rs * fix(test): review snaps & fix tests * chore(deps): update * fix(cgroup): early stop when use `map_while` * chore: init logger before setup cgroup * chore(deps): ebpf version restore to v0.12.3 * chore(lazylock): use `LazyLock::force` to init cache * fix(cache): remove unnecessary async * chore(config): make default functions const

pitoniak32 force-pushed the feat/normalize-otlp-crate branch from a307f3d to 4f44160 Compare October 19, 2024 05:14

pitoniak32 marked this pull request as ready for review October 19, 2024 12:57

pitoniak32 requested a review from a team as a code owner October 19, 2024 12:57

pitoniak32 force-pushed the feat/normalize-otlp-crate branch 3 times, most recently from 414dfac to dfeb006 Compare October 19, 2024 18:50

pitoniak32 force-pushed the feat/normalize-otlp-crate branch from dfeb006 to b4b5118 Compare October 19, 2024 21:36

cijothomas reviewed Oct 20, 2024

View reviewed changes

examples/self-diagnostics/src/main.rs Show resolved Hide resolved

cijothomas reviewed Oct 20, 2024

View reviewed changes

opentelemetry-sdk/src/metrics/exporter.rs Outdated Show resolved Hide resolved

cijothomas reviewed Oct 20, 2024

View reviewed changes

opentelemetry-stdout/examples/basic.rs Outdated Show resolved Hide resolved

cijothomas reviewed Oct 20, 2024

View reviewed changes

opentelemetry-otlp/src/exporter/mod.rs Outdated Show resolved Hide resolved

pitoniak32 force-pushed the feat/normalize-otlp-crate branch from b4b5118 to c075527 Compare October 20, 2024 13:09

pitoniak32 force-pushed the feat/normalize-otlp-crate branch 2 times, most recently from dfb21bf to f95729d Compare October 20, 2024 14:10

pitoniak32 added 2 commits October 20, 2024 10:17

feat: replace OtlpPipeline with exporter builders

b836ece

chore: remove redundant cfg feature attributes

98d3e87

these modules are conditionally included by the same flags in lib

pitoniak32 force-pushed the feat/normalize-otlp-crate branch from f95729d to 98d3e87 Compare October 20, 2024 14:17

pitoniak32 requested a review from cijothomas October 20, 2024 14:23

Merge branch 'main' into feat/normalize-otlp-crate

dbdc440

cijothomas reviewed Oct 21, 2024

View reviewed changes

examples/self-diagnostics/src/main.rs Outdated Show resolved Hide resolved

cijothomas reviewed Oct 21, 2024

View reviewed changes

opentelemetry-otlp/src/exporter/http/mod.rs Outdated Show resolved Hide resolved

cijothomas mentioned this pull request Oct 24, 2024

Unify builders across signals #2220

Merged

4 tasks

utpilla approved these changes Oct 24, 2024

View reviewed changes

cijothomas merged commit 80dc298 into open-telemetry:main Oct 24, 2024
24 of 25 checks passed

pitoniak32 deleted the feat/normalize-otlp-crate branch October 24, 2024 13:10

This was referenced Oct 24, 2024

refactor: update metrics Result to be MetricResult #2241

Merged

Refactor/metric exporters non plural #2255

Merged

refactor: InMemoryLogsExporter non plural #2258

Merged

lalitb mentioned this pull request Nov 6, 2024

fix integration test #2278

Merged

4 tasks

msuret mentioned this pull request Nov 14, 2024

feat(deps): upgrade to opentelemetry 0.27 davidB/tracing-opentelemetry-instrumentation-sdk#181

Merged

davidB pushed a commit to davidB/tracing-opentelemetry-instrumentation-sdk that referenced this pull request Nov 17, 2024

feat(deps): upgrade to opentelemetry 0.27 (#181)

8085d6d

> I followed the migration guide described in open-telemetry/opentelemetry-rust#2221

cpiemontese mentioned this pull request Nov 19, 2024

Bump the otel group with 4 updates primait/prima_tracing.rs#124

Merged

pitoniak32 mentioned this pull request Nov 22, 2024

REQUEST: New membership for pitoniak32 open-telemetry/community#2452

Closed

6 tasks

philipcristiano added a commit to philipcristiano/rust_service_conventions that referenced this pull request Dec 1, 2024

fix: otel 0.27 tls_config setting

b9319a4

open-telemetry/opentelemetry-rust#2221 Need a trait to access this now So import it

lalitb mentioned this pull request Dec 2, 2024

chore: remove the global::shutdown_tracer_provider function #2369

Merged

4 tasks

yeongjukang mentioned this pull request Dec 26, 2024

Bump the otel group with 4 updates and change builder functions kube-rs/controller-rs#124

Merged

GZTimeWalker added a commit to darkyzhou/seele that referenced this pull request Jan 5, 2025

wip: upgrade opentelemetry

09da1ec

ref: open-telemetry/opentelemetry-rust#1000 ref: open-telemetry/opentelemetry-rust#2221 ref: open-telemetry/opentelemetry-rust#2085

GZTimeWalker mentioned this pull request Jan 5, 2025

Upgrade deps and dump version to 0.6.0 darkyzhou/seele#18

Merged

gruebel mentioned this pull request Jan 25, 2025

feat: Replace Zipkin pipeline with exporter builders #2565

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: replace OtlpPipeline with exporter builders #2221

feat: replace OtlpPipeline with exporter builders #2221

pitoniak32 commented Oct 19, 2024 •

edited by cijothomas

Loading

pitoniak32 commented Oct 19, 2024 •

edited

Loading

pitoniak32 commented Oct 19, 2024

codecov bot commented Oct 19, 2024 •

edited

Loading

cijothomas commented Oct 20, 2024

cijothomas commented Oct 20, 2024

pitoniak32 commented Oct 20, 2024 •

edited

Loading

pitoniak32 commented Oct 20, 2024

cijothomas Oct 21, 2024

pitoniak32 Oct 21, 2024

cijothomas Oct 21, 2024

pitoniak32 Oct 21, 2024

pitoniak32 Oct 21, 2024

utpilla left a comment

keltia commented Nov 13, 2024 •

edited

Loading

frederikhors commented Nov 14, 2024

pitoniak32 commented Nov 14, 2024

frederikhors commented Nov 14, 2024

lalitb commented Nov 14, 2024 •

edited

Loading

frederikhors commented Nov 14, 2024 •

edited

Loading

lalitb commented Nov 14, 2024 •

edited

Loading

frederikhors commented Nov 15, 2024

lalitb commented Nov 15, 2024

cijothomas commented Nov 15, 2024

feat: replace OtlpPipeline with exporter builders #2221

feat: replace OtlpPipeline with exporter builders #2221

Conversation

pitoniak32 commented Oct 19, 2024 • edited by cijothomas Loading

Migration Guide

Exporter Guide

Provider Guide

Signal Examples

Trace Example

Metric Example

Log Example

Changes

Merge requirement checklist

pitoniak32 commented Oct 19, 2024 • edited Loading

pitoniak32 commented Oct 19, 2024

codecov bot commented Oct 19, 2024 • edited Loading

Codecov Report

cijothomas commented Oct 20, 2024

cijothomas commented Oct 20, 2024

pitoniak32 commented Oct 20, 2024 • edited Loading

pitoniak32 commented Oct 20, 2024

cijothomas Oct 21, 2024

Choose a reason for hiding this comment

pitoniak32 Oct 21, 2024

Choose a reason for hiding this comment

cijothomas Oct 21, 2024

Choose a reason for hiding this comment

pitoniak32 Oct 21, 2024

Choose a reason for hiding this comment

pitoniak32 Oct 21, 2024

Choose a reason for hiding this comment

utpilla left a comment

Choose a reason for hiding this comment

keltia commented Nov 13, 2024 • edited Loading

frederikhors commented Nov 14, 2024

pitoniak32 commented Nov 14, 2024

frederikhors commented Nov 14, 2024

lalitb commented Nov 14, 2024 • edited Loading

frederikhors commented Nov 14, 2024 • edited Loading

lalitb commented Nov 14, 2024 • edited Loading

frederikhors commented Nov 15, 2024

lalitb commented Nov 15, 2024

cijothomas commented Nov 15, 2024

pitoniak32 commented Oct 19, 2024 •

edited by cijothomas

Loading

pitoniak32 commented Oct 19, 2024 •

edited

Loading

codecov bot commented Oct 19, 2024 •

edited

Loading

pitoniak32 commented Oct 20, 2024 •

edited

Loading

keltia commented Nov 13, 2024 •

edited

Loading

lalitb commented Nov 14, 2024 •

edited

Loading

frederikhors commented Nov 14, 2024 •

edited

Loading

lalitb commented Nov 14, 2024 •

edited

Loading