Add singleton flags to factories and standardize attributes

[djaglowski]

This PR is an extension of #11344 that addresses #11814.

Major changes:

• Add a Metadata() method to receiver.Factory, exporter.Factory and connector.Factory (processor.Factory excluded intentionally as described in Loggers for shared components should not report signal type. #11814). These Metadata() methods return Metadata structs, which act similarly to consumer.Capabilities in that they are intended as a set of information which describes the components which the factory produces.
• Remodels each component's as a proper go.opentelemetry.io/otel/attribute.Set. This set is used as the ID in the graph, but also as the set of attributes that creates the logger. In the future, the set will be automatically used to describe other aspects of the collector's own telemetry.
• The logger fields are changed to match those described in the component telemetry RFC.

Minor changes:

• Unknown component types are detected earlier in the startup process. (When the graph is being assembled, rather than when the components of the graph are being instantiated.) This is necessary so that we can obtain the Metadata from the appropriate factory. There is a minor user facing change in that the error message is not wrapped in a "could not build" prefix. I moved the processors to this pattern as well in order to match the other components.

[djaglowski]

If desired, I believe I can split this PR into several smaller PRs. However, I wanted to put this version up in order to show the overall picture.

[codecov]

Codecov Report

Attention: Patch coverage is 98.31224% with 4 lines in your changes missing coverage. Please review.

Project coverage is 91.88%. Comparing base (dcc866c) to head (02f5e8f).
Report is 62 commits behind head on main.

Files with missing lines	Patch %	Lines
service/internal/graph/exporter.go	92.59%	2 Missing ⚠️
service/internal/graph/receiver.go	95.23%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #12057      +/-   ##
==========================================
+ Coverage   91.72%   91.88%   +0.15%     
==========================================
  Files         463      465       +2     
  Lines       24821    25430     +609     
==========================================
+ Hits        22767    23366     +599     
- Misses       1672     1675       +3     
- Partials      382      389       +7     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[djaglowski]

I discovered when adding tests that singleton connectors will introduce broader changes into the way we build the graph, since we're wrapping each instance as a specific type of consumer. I've not been able to get it working yet but either way, the additional complexity seems not worth for now. I believe there are some reasons why singleton connectors would be useful but I'm going to propose for now that singleton instances are supported only for receivers and exporters. In the future we can try to add singleton connectors.

[mx-psi]

The approach makes sense to me, just one question about naming

[mx-psi]

Looks good to me.

One thing I don't like about Capabilities is that a component developer can accidentally 'lie' about their component, which can also happen here. I don't see a great way to do this here without introducing a lot more API, so this seems reasonable enough

[jade-guiton-dd]

Do you have ideas for making the new attributes available to components for use in spans and metrics? (Adding the attribute.Set in the TelemetrySettings for example?)

[djaglowski]

Do you have ideas for making the new attributes available to components for use in spans and metrics? (Adding the attribute.Set in the TelemetrySettings for example?)

It is the next thing I intend to look into once this is merged. Either as you suggested, or maybe it's possible to save them into the Meter/Tracer providers in some way?

[mx-psi]

I will merge this on Monday unless somebody comments, cc @open-telemetry/collector-approvers

[songy23]

LGTM

[mx-psi]

@dmitryax will take a look at this before we merge it

[sfc-gh-bdrutu]

I have a feedback we discuss in the otel collector stability meeting (which may be already address, but want to make sure is documented): We should be able to have internal telemetry (logs/traces/metrics) annotated with the signal type that produces them, so if I fail to parse a message I should know that is the "metrics" pipeline that failed.

PS: What I think we discussed is exactly against #11814.

I do understand that from a "status" perspective if the main shared gRPC server fails is a fail of all components, but if a message is logged when I process the logs I should know that is about logs.

[djaglowski]

I have a feedback we discuss in the otel collector stability meeting (which may be already address, but want to make sure is documented): We should be able to have internal telemetry (logs/traces/metrics) annotated with the signal type that produces them, so if I fail to parse a message I should know that is the "metrics" pipeline that failed.

PS: What I think we discussed is exactly against #11814.

I do understand that from a "status" perspective if the main shared gRPC server fails is a fail of all components, but if a message is logged when I process the logs I should know that is about logs.

I mostly agree with you but I think we need to look at this as two layers:

1. Provide a set of attributes which identifies the instance, and also is correct when describing the instance as a whole.
2. Encourage (or enforce in some way) that signal-specific telemetry is additionally attributed to the signal type.

I think this PR solves (1) and allows for (2) to be a best/recommended practice. (e.g. otlp receiver has a logger w/o signal attribute, but should use zap.String("otel.signal", "logs") where appropriate.

We can consider ways to make (2) automatic/natural later, but I think this PR is a big improvement over the current state of things where we incorrectly attribute all logs to the same signal (whichever was created first).

[bogdandrutu]

I do understand the underlying problem you try to solve, but I think this will come with the downside that the signal specific part of the component is not handled correctly (or has to be handled by manually annotating the signal tag to the telemetry).

In reality we don't need to share the entire component, but we need to share a "sub-component" which indeed is signal agnostic, for example the "http.Server" part of a receiver. This sub-component is then wrapped in signal specific logic.

The current approach is to treat the entire component as shared and not have the Signal annotation available, probably later will manually add that (this is my expectation since I don't see any other way to do this).

I would rather go with a different approach which will be to add a concept of a "core/share" component to the factory.

[djaglowski]

I'm curious to hear more about your idea but my immediate thoughts:

1. You cannot effectively prevent component authors from making the whole component a singleton. So we might as well allow it and work with it. (Maybe could detect and fail when CreateX and CreateY return the same struct, but this can always be subverted by adding one more layer of abstraction.)
2. While I would also like to eliminate correctness concerns, omitting a signal attribute when it could have been included is a trivial problem to fix. It won't ever be implemented perfectly across all components but will trend upward very quickly.
3. What you're proposing will require substantial refactoring to several important components, each of which will have unique design changes and corresponding correctness challenges. I doubt you or I have the time to do this and even if we did it would delay the addition of several useful observability signals from the collector by additional months.

[djaglowski]

I think this will come with the downside that the signal specific part of the component is not handled correctly (or has to be handled by manually annotating the signal tag to the telemetry).

I agree this is a downside, but your proposal leaves the same problem in place. It reduces the scope of the problem slightly at the cost of a lot of refactoring, but fundamentally the same problem remains. Any shared sub-component may have good reason to report telemetry that pertains to one specific signal, so how does the shared sub-component manage this? This still leaves us with manually annotating the signal tag to the telemetry.

[dmitryax]

What if we just add another field to component.TelemetrySettings, let's call it SharedLogger (same as Logger but without the signal attribute), which can be used by the shared parts of the components while Logger will be kept to signal specific code parts? memory_limiter processor needs that as well.

It's hard to be convinced that this new public API (Metadata) is required. It feels like just passing a boolean flag back to the factory. It's supposed to be connected to the fact that sharedComponent is used, but there is nothing to support that connection.

[djaglowski]

I'm closing this as I no longer believe that adding notions of shared/singleton parts is a realistic solution to all the novel cases. New proposal here: #12217