What is the difference between the API and SDK?

[emschwartz]

As a user of opentelemetry, there are quite a few terms to learn when navigating the various packages to achieve some goal.

One of the things I've found confusing is the separation between the opentelemetry_api and opentelemetry_sdk. As an outsider that may be missing some project-specific context, those two terms sound like they ought to be interchangeable. My general understanding of SDKs are that they're libraries or sets of libraries that help you build against or on top of APIs. I don't understand why they're separate in OpenTelemetry world and it always feels somewhat random to me what things I need to import from the API or SDK crate.

The OpenTelemetry glossary doesn't help that much:

Application Programming Interface. In the OpenTelemetry project, used to define how telemetry data is generated per Data Source.

Short for Software Development Kit. Refers to a telemetry SDK that denotes a Library that implement the OpenTelemetry API.

Could this be clarified further in the docs or (I assume not) could the crates just be combined?

[cijothomas]

See if this spec doc helps clarify this more : https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/library-guidelines.md

API is what is required to "instrument" - describe the shape of telemetry - like name of span, its attributes, metric values, etc.
The API has no notion of things like "How to sample telemetry", or "how to export telemetry".
API has a "no-op" implementation, so span/metrics etc. are "no-ops" unless an SDK is installed, which lights up actual telemetry flow.
Libraries MUST only depend on the API as they only need to describe the shape of telemetry (and has no need to know where the telemetry is exported to finally)

SDK implements the API (replacing no-ops with actual implementation), and also deals with concepts of sampling, exporting telemetry to backends etc.

The idea behind API and SDK being separate is : One is free to replace the official SDK with their own custom implementation, without requiring re-instrumenting of the app/library. API has higher stability guarantees, as any change to API requires re-instrumenting. If SDK has breaking changes, it only affects a smaller portion of the applications as setting up SDK is typically a one-time operation done at app startup, as opposed to API calls which could be spread throughout the applications (and thus could be very costly to re-instrument)

[djc]

I think the concepts make sense, but (at least for me) the naming difference is quite subtle/confusing. At least I haven't seen this naming used before in the Rust ecosystem.

[emschwartz]

I see. Thanks a lot @cijothomas for the explanation. I agree with @djc that it does make sense but the difference is quite subtle and maybe a bit hard for Rust developers coming to this library/ecosystem.

[emschwartz]

One thing that might help (though I understand this would be a big change, so I don't expect it to happen) would be to make the opentelemetry crate merge the things from the API and SDK together. Instead of having use opentelemetry::sdk::metrics::... and use opentelemetry::metrics..., you would just have use opentelemetry::metrics::....

The opentelemetry_api and _sdk crates exist separately for anyone that wants to use them separately, and the opentelemetry crate can make it so a new user doesn't need to understand this distinction. The opentelemetry crate automatically pulls in both the API and SDK so it only makes sense to use if you want both anyway.

[shaun-cox]

Currently, I think the opentelemetry crate is somewhat of a backward compatibility catch-all to be able to easily use all types whether they belong to the api or the sdk, which are well-defined (and important) distinctions called out in the OpenTelemetry Client Architecture as @cijothomas mentions.

The issue with having the opentelemetry crate name be the "all-in-one" or "kitchen sink" crate, and not just be the interface to OpenTelemetry is that it promotes false dependencies.

There is also a bit of precedent with having the root crate for an ecosystem just be the interface... take tracing as an example. The tracing crate is not the "all-in-one" crate... it explicitly states it is just the API.

While my preference would be for opentelemetry to contain only the API, others have expressed a desire to be explicit, so with that in mind, I propose the following goal:

• promote opentelemetry_api as the crate to depend on for the interface
• promote opentelemetry_sdk as the crate to depend on for the sdk
• deprecate the opentelemetry crate

I think the immediate first step is to examine reverse dependencies of opentelemetry crate and start changing those to depend on either, or both of opentelemetry_api and opentelemetry_sdk.

[hdost]

I think after creating this initial commit on tracing-opentelemetry, it's not to terrible to refactor.

Also I dug up the old issue #721, even then though I was definitely more in favor of opentelemetry being the API.
The biggest reason I think is discoverability, it's more ideomatic in crates that when you see a crate with the name opentelemetry that's the one you should go for.

[lalitb]

I think after creating this initial commit on tracing-opentelemetry, it's not to terrible to refactor.

This looks good. Probably also move opentelemetry-sdk as dev-dependency if it is only used in examples and tests.

[TommyCpp]

it's more ideomatic in crates that when you see a crate with the name opentelemetry that's the one you should go for

That's a good point. I wonder in that case should we just make opentelemtry a re-export of opentelemetry-api.

[djc]

That's a good point. I wonder in that case should we just make opentelemtry a re-export of opentelemetry-api.

Seems reasonable.

I wonder if there's a better/more descriptive name we can think of for _sdk, though.

[hdost]

@jtescher Figured it'd be good to get you to chime in here.

[shaun-cox]

Similar to @hdost above, I have draft PR (#1199) showing would-be changes if the proposal above is accepted.

[jtescher]

I think the point that is most difficult to make a "great" DX is that the API and SDK packages have to be separated as the intention of otel is to allow other third party SDK implementations. But given that we have to have two packages, I think whatever is clearest that one is the "main" package that all apps and libraries should use, and the other is the "chosen sdk impl that the app has specified and libraries should not rely on".

In that sense having the main api be opentelemetry and the sdk being opentelemetry-sdk makes sense to me, but I don't feel strongly about the specific names and more that they convey the concept as clearly as we can.

[cijothomas]

My vote would be to name crates explicitly to indicate purpose - opentelemetry_api and opentelemetry_sdk should be the names, and there should be no need for a "opentelemetry" crate, as it is not clear what it is referring to. If there is a desire to make a uber crate bringing a bunch of commonly used crates, it could be named "opentelemetry-starter" or "opentelemetry-all-in-one" or similar to make the intention clear.

OpenTelemetry .NET called their SDK as simply "OpenTelemetry" (there is almost never a need for anyone to install SDK separately, as Exporters automatically bring the SDK), and API as OpenTelemetry.API. But almost every other language implementations call API as "opentelemetry-api" and SDK as "opentelemetry-sdk", with nothing called just 'opentelemetry'.

Java : https://search.maven.org/search?q=g:io.opentelemetry
https://opentelemetry.io/docs/instrumentation/python/#installation
https://www.nuget.org/profiles/OpenTelemetry

[shaun-cox]

I'll continue with the PR above then to ready it for merge.

[hdost]

An additional argument for opentelemetry being used to is that aside from search is that right now there's a lot of talk about squatting and names/namespacing related to crates.io and it's again because of the convention the Rust community typically follows that the crate everyone must use is X and the additional crates are X-foo.

I'll continue with the PR above then to ready it for merge.

Might want to hold until we come to a concensus.

it's more ideomatic in crates that when you see a crate with the name opentelemetry that's the one you should go for

That's a good point. I wonder in that case should we just make opentelemtry a re-export of opentelemetry-api.

We could do that or the reverse it should work either direction.

[lalitb]

Though not a good comparison with otel-cpp as we don't release any binary packages, but only source distribution. we segregate api and sdk in separate namespaces. So for trace signal,

• api is under opentelemetry::trace namespace
• sdk is under opentelemetry::sdk::trace namespace
• exporters under opentelemetry::exporter::otlp::trace namespace

So no "api" prefix/naming for API code. And C++ community is in general used-to with this convention.

My thoughts are - The naming of these packages should adhere to the conventions commonly observed within the language community. As these packages are intended for Rust developers, they should reflect the general ideology and principles of how things operate within the Rust ecosystem.

[cijothomas]

Given the established pattern in Rust community, I think I'll change my view - "opentelemetry" crate can be the API (similar to how tracing, log crates are just the facade in already established crates)

[shaun-cox]

Cool. With that, I'll change the proposal to:

• promote opentelemetry as the crate to depend on for the interface
• promote opentelemetry_sdk as the crate to depend on for the sdk
• remove the re-export of opentelemetry_sdk types from opentelemetry

Working on the PR today.

[TommyCpp]

Reopen as we have other PRs related to this effort