capture code.stacktrace and mutable spans

[SylvainJuge]

Hi,

We currently have a feature in our OTel agent distribution that allows to capture the code.stacktrace (semconv) to show where the span has been created. This is something that we'd like to contribute as a standalone SDK extension in the opentelemetry-java-contrib.

There are however a few implementation details that make this challenging:

• the SDK SpanProcessor interface only allows to modify span attributes when the span is started
• capturing stack traces on every span can be expensive, so doing it only for spans that have a minimal duration seems relevant, however the span duration is only available on span end
• capturing the stack trace on span end is usually a good approximation of where it has been started, and that's usually an acceptable compromise.
• there is currently no simple way to modify the span on end and the topic has been raised few times already:
  • Add BeforeEnd to have a callback where the span is still writeable opentelemetry-specification#1089
  • Ability to add span attributes in SpanProcessor's onEnd operation opentelemetry-specification#2990

In order to do this, we need to have a way to get a mutable span (MutableSpan.java) that allows to add extra attributes. One of the reasons the SDK does not provide mutable spans is that the order of the span processors can't be guaranteed (see comment) and so combining multiple span processors can produce unexpected results based on their registration order.

To remove this limitation and allow proper usage or mutable spans, we also had to introduce an explicit way to order the span processors through a "processor chain" that is similar to the servlet filter chains. The implementation can be found in AbstractSimpleChainingSpanProcessor.java and it even supports autoconfiguration.

We plan to contribute in a few steps:

• adding support for mutable spans (1)
• adding support for chained span processors (2)
• adding the span processor that captures code.stacktrace (3)

The questions are the following:

• do you think contributing this feature is valuable ?
• if yes, then I suggest creating a new span-stacktrace sub-project in opentelemetry-java-contrib and being one of the owners of it.
• (1) and (2) are just utilities, where should we contribute them ? Should we keep them in a separate sub-project of the span-stacktrace as a first step and maybe later provide a dedicated top-level sub-project for it?

[SylvainJuge]

Contributed in open-telemetry/opentelemetry-java-contrib#1255 with using a chaining span processor while waiting for a better solution with changes in the SDK.