Add SDK span telemetry metrics

[JonasKunz]

Changes

With this PR I'd like to start a discussion around adding SDK self-monitoring metrics to the semantic conventions.
The goal of these metrics is to give insights into how the SDK is performing, e.g. whether data is being dropped due to overload / misconfiguration or everything is healthy.
I'd like to add these to semconv to keep them language agnostic, so that for example a single dashboard can be used to visualize the health state of all SDKs used in a system.

We checked the SDK implementations, it seems like only the Java SDK currently has some health metrics implemented.
This PR took some inspiration from those and is intended to improve and therefore supersede them.

I'd like to start out with just span related metrics to keep the PR and discussions simpler here, but would follow up with similar PRs for logs and traces based on the discussion results on this PR.

Prior work

This PR can be seen as a follow up to the closed OTEP 259:

• The OTEP originally superseded Add processed/exported Span metrics. #184, which initially focused only on SDK exporter metrics.
• Add processed/exported Span metrics. #184 was closed in favor of the predecessor of OTEP 259 to instead allow monitoring of entire "pipelines" using unified metrics across SDK exporters and collector components : A pipeline starts with an SDK exporter and goes through the processing of collector(s)
• Finally, that OTEP was closed in favor of this collector RFC which adds metrics just for the collector components.

So we kind of have gone full circle: The discussion started with just SDK metrics (only for exporters), going to an approach to unify the metrics across SDK-exporters and collector, which then ended up with just collector metrics.
So this PR can be seen as the required revival of #184 (see also this comment).

In my opinion, it is a good thing to separate the collector and SDK self-metrics:

• There have been concerns about both using the same metrics for both: How do you distinguish the metrics exposed by collector components from the self-monitoring metrics exposed by an Otel-SDK used in the collector for e.g. tracing the collector itself?
• Though many concepts between the collector and SDK share the same name, they are not the same thing (to my knowledge, I'm not a collector expert): For example processors in the collector are designed to form pipelines potentially mutating the data as it passes through. In contrast, SDK span processor don't form pipelines (at least not visible to the SDK, those would be hidden custom implementations). Instead SDK span processors are merely observers with multiple callbacks for the span lifecycle. So it would feel like "shoehorning" things into the same metric, even though they are not the same concepts.
• Separating collector and SDK metrics makes their evolution and reaching agreements a lot easier: When using separate metrics and namespaces, collector metrics can focus on the collector implementation and SDK metrics can be defined just using the SDK spec. If combine both in shared metrics, those will have to be always be aligned with both the SDK spec and the collector implementation. I think this would make maintenance much harder for little benefit.
• I have a hard time finding benefits of sharing metrics for SDK and collector: The main benefit I find would of course be easier dashboarding / analysis. However, I do think having to look at two sets of metrics to do so is a fine tradeoff, considering the difficulties with the unification listed above and shown by the history of OTEP 259.

Existing Metrics in Java SDK

For reference, here is what the existing health metrics currently look like in the Java SDK:

Batch Span Processor metrics

• Gauge queueSize, value is the current size of the queue
  • Attribute spanProcessorType=BatchSpanProcessor (there was a former ExecutorServiceSpanProcessor which has been removed)
  • This metric currently causes collisions if two BatchSpanProcessor instances are used
• Counter processedSpans, value is the number of spans submitted to the Processor
  • Attribute spanProcessorType=BatchSpanProcessor
  • Attribute dropped (boolean), true for the number of spans which could not be processed due to a full queue

The SDK also implements pretty much the same metrics for the BatchLogRecordProcessor just span replaced everywhere with log

Exporter metrics

Exporter metrics are the same for spans, metrics and logs. They are distinguishable based on a type attribute.
Also the metric names are dependent on a "name" and "transport" defined by the exporter. For OTLP those are:

• exporterName=otlp
• transport is one of grpc, http (= protobuf) or http-json

The transport is used just for the instrumentation scope name: io.opentelemetry.exporters.<exporterName>-<transport>

Based on that, the following metrics are exposed:

• Counter <exporterName>.exporter.seen: The number of records (spans, metrics or logs) submitted to the exporter

  • Attribute type: one of span, metric or log

• Counter <exporterName>.exporter.exported: The number of records (spans, metrics or logs) actually exported (or failed)

  • Attribute type: one of span, metric or log
  • Attribute success (boolean): false for exporter failures

Merge requirement checklist

• CONTRIBUTING.md guidelines followed.
• Change log entry added, according to the guidelines in When to add a changelog entry.
  • If your PR does not need a change log, start the PR title with [chore]
• schema-next.yaml updated with changes to existing conventions.

[lmolkova]

Related #1580

[reyang]

Do we allow each language implementations to have additional attributes that are language specific?

[JonasKunz]

I don't see a reason why implementations shouldn't be allowed to add additional attributes. I would expect this to be the general case for all semconv metrics? Metrics are aggregateable, so they can be analyzed and presented as if those additional attributes weren't present.

There are two caveats I can think of:

• The metrics are recommended to be enabled by default. Therefore they must have a very, very low cardinality to justify this and not cause to much overhead. So depending on the cardinality of the additional attributes, they should probably be opt-in.
• The attributes might conflict with future additions to the spec, so you'll end up with breaking changes. So best to use some language-specific attribute naming.

[lmolkova]

I think we should include recommended server.address and server.port attributes on exporter metrics. It's good to know where you are sending data to.

[dashpole]

Those would not apply to all exporters (e.g. stdout). My thinking is that we should encourage using protocol-level instrumentation (e.g. http/gRPC) for details like this.

[JonasKunz]

Kind of agree with @dashpole here. I don't think this belongs in this metric.
Nonetheless, I think it would make sense to add exporter.request.* metrics to track request stats (e.g. bytes sent, response codes, server details). However, I don't think that this should happen in this PR, but rather in a separate, follow-up PR. It is an enhancement to gain more fine grained insights in addition to the metrics to this PR, but doesn't have an impact on them.

[lmolkova]

Those would not apply to all exporters (e.g. stdout).

not a problem, just add them with requirement level recommended: when applicable. We do include these attributes on logical operations across semconv, so they do belong here.

[JonasKunz]

Added in 830edfb.

However, for attribute references for metrics there is no schema.compliant recommended: when applicable IINM.
I added note: recommended when applicable instead, please let me know if this is the correct approach.

[dashpole]

I completed the Go prototype of the proposed semantic conventions: open-telemetry/opentelemetry-go#6153