[consumer] Allow annotating consumer errors with metadata #9041

evan-bradley · 2023-12-05T03:04:02Z

Description:

Revival of #7439

This explores one possible way to allow adding metadata to errors returned from consumers. The goal here is to allow transmitting more data back up the pipeline if there is an error at some stage, with the goal of it being used by an upstream component, e.g. a component that will retry data, or a receiver that will propagate an error code back to the sender.

The current design eliminates the permanent/retryable error types in favor of a single error type that supports adding signal data to be retried. If no data is added to be retried, the error is considered permanent. Currently there is no distinction made between the signals for the sake of simplicity, the caller should know what signal is used when retrieving the retryable items from the error. Any options for retrying the data (e.g. a delay) are offered as options when adding data to retry.

The error type currently supports a few general metadata fields that are copied when a downstream error is wrapped:

Partial successes can be expressed by setting the number of rejected items.
gRPC and HTTP status codes can be set and translated between if necessary.

Link to tracking Issue:

Resolves #7047

cc @dmitryax

consumer/consumererror/README.md

consumer/consumererror/consumererror.go

jmacd · 2024-01-10T19:11:16Z

Happy to see this added. As discussed in #9260, there is a potential to propagate backwards the information contained in PartialSuccess responses from OTLP exports.

I worry about the code complexity introduced to have "success error" responses, meaning error != nil but the interpretation being success. However, this is what it will take to back-propagate partial successes, we want callers to see success with metadata about the number of rejected points if possible. Great to see this, thanks @evan-bradley.

jmacd · 2024-01-10T19:12:19Z

As discussed in open-telemetry/oteps#238, it would be useful for setting the correct otel.outcome label, for callers to have access to the underlying gRPC and/or HTTP error code. Thanks!

consumer/consumererror/consumererror.go

consumer/consumererror/partial.go

github-actions · 2024-02-22T03:15:22Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

github-actions · 2024-03-21T03:15:44Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

consumer/consumererror/internal/statusconversion/conversion.go

consumer/consumererror/statuserrors.go

codecov · 2024-04-11T18:34:33Z

Codecov Report

Attention: Patch coverage is 98.70130% with 1 line in your changes missing coverage. Please review.

Project coverage is 91.67%. Comparing base (ecbe02e) to head (1d1affc).

Files	Patch %	Lines
consumer/consumererror/error.go	98.03%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #9041      +/-   ##
==========================================
+ Coverage   91.64%   91.67%   +0.02%     
==========================================
  Files         406      408       +2     
  Lines       19001    19078      +77     
==========================================
+ Hits        17414    17490      +76     
- Misses       1227     1228       +1     
  Partials      360      360

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

consumer/consumererror/statuserrors.go

consumer/consumererror/internal/statusconversion/conversion.go

consumer/consumererror/partial.go

consumer/consumererror/README.md

bogdandrutu · 2024-07-30T17:48:40Z

consumer/consumererror/README.md

+- `consumererror.WithRetry`
+- `consumererror.WithPartial`


Let's assume is retryable, what data are you retrying? Based on my understanding "partial" only contains the number of failed entries, so you cannot retry because you don't know what failed.

bogdandrutu · 2024-07-30T17:50:31Z

consumer/consumererror/README.md

+**Retryable**: Errors are retryable if re-submitting data to a sink may result
+in a successful submission.


This needs to carry some informations:

[required] what to retry on;

[optional] retry interval?

This is just a classification of error, I cover the details of what retryable errors contain here: https://github.com/open-telemetry/opentelemetry-collector/pull/9041/files#diff-e0fa8222784a2c0c2683b70e2b6a7ccf2c54b6477acd7e2518839e38060bdcf5R90.

We discussed this last week and determined that the caller can provide the data to retry, but we're going to focus on exactly what goes into a retryable error after this PR. Right now we're mainly focusing on the HTTP/gRPC errors.

bogdandrutu · 2024-07-30T17:51:24Z

consumer/consumererror/README.md

+**Permanent**: Errors are permanent if submission will always fail for the
+current data. Errors are considered permanent unless they are explicitly marked
+as retryable.


For permanent errors, we should probably always return the "number of not accepted" entries? Is that reasonable?

I don't think we need to do that since the caller will already know how many items failed.

bogdandrutu · 2024-07-30T17:54:04Z

consumer/consumererror/README.md

+**Communicating network error codes**: Errors should allow embedding information
+necessary for the Collector to act as a proxy for a backend, i.e. relay a status
+code returned from a backend in a response to a system upstream from the
+Collector.


I think @jpkrohling had a more detailed use-case here, but would be interesting to hear what problem we are solving. If I remember correctly was something that the collector should not retry and all retries to be done by the caller, but cannot remember exactly, so better to document in details here.

I can provide more examples if you like, but the simplest one is the one I described here: the Collector can act as a proxy and relay a code from a backend back to the caller. This works for retries too, if the code is e.g. an HTTP 429 status code.

consumer/consumererror/README.md

bogdandrutu · 2024-07-30T17:58:54Z

consumer/consumererror/README.md

+- `consumererror.WithPartial`
+- `consumererror.WithGRPCStatus`
+- `consumererror.WithHTTPStatus`
+- `consumererror.WithMetadata`


I am not seeing any use-case for this, why do we need it?

This is for extensibility. I don't think we would add this until it was requested, but basically a custom exporter and receiver could communicate using a custom struct type by putting the struct inside the error with WithMetadata.

Maybe remove it for now? We should problably ensure the model is extendable but not over-prescriptive for something we don't have yet

If extensibility is an explicit goal, I'd like to keep this in the design doc. I've added warnings to things that don't actually exist right now.

bogdandrutu · 2024-07-30T17:59:49Z

consumer/consumererror/README.md

+- `WithRetry` and `WithPartial` are included together: Partial successes are
+  considered permanent errors in OTLP, which conflicts with making an error
+  retryable by including `WithRetry`. However, per our definition of what makes
+  a permanent error, this error has been marked as retryable, and therefore we
+  assume the component producing this error supports retyable partial success
+  errors.


See my previous question about this and what it means if I know that 5 out of 10 spans failed to send.

If I know that 5/10 spans failed, then I can record that in obsreport, but will resubmit all 10; maybe the backend uses the span IDs to make span submission idempotent. I don't have a real use-case in mind for this, what I want to cover in this section is that we can use the rules established here to handle seemingly invalid error combinations.

I don't have a real use-case in mind for this, what I want to cover in this section is that we can use the rules established here to handle seemingly invalid error combinations.

When in doubt leave it out. I want us to not add any API if we don't have a good use-case.

Agreed. This doesn't add any API, the point of this paragraph is just to explain how seemingly conflicting error combinations could be reconciled. WithRetry and WithPartial are intended to be used separately, and already have use cases in mind. We just needed to explain what happens if a component author attempts to use them together.

dmitryax

Should we briefly define a transition plan in the doc?

consumer/consumererror/README.md

dmitryax · 2024-08-05T16:44:30Z

consumer/consumererror/README.md

+- `consumererror.WithPartial`
+- `consumererror.WithGRPCStatus`
+- `consumererror.WithHTTPStatus`
+- `consumererror.WithMetadata`


Maybe remove it for now? We should problably ensure the model is extendable but not over-prescriptive for something we don't have yet

consumer/consumererror/error.go

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

evan-bradley · 2024-08-05T20:33:35Z

Should we briefly define a transition plan in the doc?

I'm okay with that. I've put it in a section at the bottom for now.

evan-bradley · 2024-08-08T21:29:40Z

@dmitryax @bogdandrutu This is ready for another round of reviews. The scope should be fairly minimal now, we can cover additional functionality in follow-ups.

I will be on PTO starting next week, so it would be nice if we could get this in this week.

bogdandrutu

Overall feedback for me is that we just want to add more "things". I don't want to block this, but I also feel that we should make sure we add only the things we are 100% sure about it.

bogdandrutu · 2024-08-09T03:40:59Z

consumer/consumererror/README.md

+- `WithRetry` and `WithPartial` are included together: Partial successes are
+  considered permanent errors in OTLP, which conflicts with making an error
+  retryable by including `WithRetry`. However, per our definition of what makes
+  a permanent error, this error has been marked as retryable, and therefore we
+  assume the component producing this error supports retyable partial success
+  errors.


I don't have a real use-case in mind for this, what I want to cover in this section is that we can use the rules established here to handle seemingly invalid error combinations.

When in doubt leave it out. I want us to not add any API if we don't have a good use-case.

bogdandrutu

Overall:

I really like the MultiErr (current Error) approach and the APIs related to that. It is a nice way to encapsulate fanout errors.
I am not sure if the ErrorData is the right thing though. Is this only for "Transport" errors? I would not return that bool but instead have a enum or a way to say "IsHttp" or "isGrpc" then have "AsHttp" or "AsGrpc" name to be discussed.

bogdandrutu · 2024-08-09T03:46:21Z

consumer/consumererror/error.go

+// out by an upstream component by calling `Error.Data`.
+//
+// Experimental: This API is at the early stage of development and may change without backward compatibility
+type ErrorData interface {


What is the point of making it interface vs struct?

The goal is that component authors can't instantiate it themselves. If we're not concerned about that, I don't see any issues making it a struct.

consumer/consumererror/error.go

bogdandrutu · 2024-08-09T03:49:36Z

consumer/consumererror/error.go

+}
+
+// ErrorOption allows annotating an Error with metadata.
+type ErrorOption func(error *errorData)


Not recommended to have a public type that wraps a private type. I think the best way to do this is to use the interface pattern, see

opentelemetry-collector/receiver/internal/factory.go

Line 60 in ecbe02e

type FactoryOption interface {

I think I like this a bit better, thanks. Implemented.

bogdandrutu · 2024-08-09T03:50:25Z

consumer/consumererror/error.go

+	errors []ErrorData
+}
+
+// ErrorData is intended to be used to encapsulate various information


Would call this "Error" and the other "Errors"(or ErrorSlice)?

Maybe "MultiErr" for the container? Not sure, but throwing ideas.

I've renamed them to ErrorContainer (for the type that accumulates errors as they move upstream) and Error for the actual error with the data. What do you think?

bogdandrutu · 2024-08-09T03:53:26Z

consumer/consumererror/error.go

+// Data returns all the accumulated ErrorData errors
+// emitted by components downstream in the pipeline.
+// These can then be aggregated or worked with individually.
+func (e *Error) Data() []ErrorData {


This is wrong I think, we should not return the internal slice since the caller can modify it.

From what I can tell, this creates a new slice instance where modifications don't affect the original slice.

I added a test here that seems to show this isn't an issue, is there anything else I should check for to ensure we don't pass out a mutable copy of the slice? https://github.com/open-telemetry/opentelemetry-collector/pull/9041/files#diff-8c40d97e3ff1a76b10c0c2b02a2cea669f1ffdffc2363cfdf86ba8489cb9f59cR92

Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>

evan-bradley · 2024-08-09T12:42:34Z

Overall feedback for me is that we just want to add more "things". I don't want to block this, but I also feel that we should make sure we add only the things we are 100% sure about it.

Right now we're only adding HTTP/gRPC status code propagation, everything else is marked as unimplemented and still in the design stage. Do you have concerns with HTTP/gRPC status codes, or the way they're communicated?

I really like the MultiErr (current Error) approach and the APIs related to that. It is a nice way to encapsulate fanout errors.

I played with that, but the issue I encountered is that we may want to read these error types at the receiver level, when there may be multiple fanouts in the pipeline. So you could see something like the following:

~~I don't see how we pull out the joined errors without traversing the whole error tree by repeatedly calling Unwrap. Having a container type to keep track of all the errors just seems simpler.~~

I see now, I mistook what you said for the Uber multierr package. Glad you like it.

I am not sure if the ErrorData is the right thing though. Is this only for "Transport" errors? I would not return that bool but instead have a enum or a way to say "IsHttp" or "isGrpc" then have "AsHttp" or "AsGrpc" name to be discussed.

This is meant to hold information related to anything the component would like to communicate upstream. Right now we're starting with HTTP and gRPC status codes, but we would like to add retry information, partial success information, etc. in the future. The goal is to start small and iterate as we design the additional options.

bogdandrutu · 2024-08-09T16:16:09Z

This is meant to hold information related to anything the component would like to communicate upstream. Right now we're starting with HTTP and gRPC status codes, but we would like to add retry information, partial success information, etc. in the future. The goal is to start small and iterate as we design the additional options.

Can you please respond to the rest of the comments and see how the design of the "ErrorData" would look like?

evan-bradley · 2024-08-09T16:31:28Z

Can you please respond to the rest of the comments and see how the design of the "ErrorData" would look like?

Sure thing. The other questions were based on the first question, so I wanted to make sure we were on the same page there if possible.

I am not sure if the ErrorData is the right thing though. Is this only for "Transport" errors? I would not return that bool but instead have a enum or a way to say "IsHttp" or "isGrpc" then have "AsHttp" or "AsGrpc" name to be discussed.

Enum: We could do a list of enums, e.g. [IS_RETRYABLE, IS_GRPC], then expose this with a method like Types() []Type. Since an error can contain multiple error types this would have to be a list instead of just a single enum value.
IsHTTP + AsHTTP: This essentially conveys the same information as HTTPStatus() (code int, isHTTP bool), so from a functionality standpoint it's not an issue. I would rename IsHTTP to HasHTTP though, since errors can contain multiple types of data. I'm not sure I see the benefit of retrieving this information across two methods, though, what's wrong with the boolean?

bogdandrutu · 2024-08-11T18:06:14Z

Enum: We could do a list of enums, e.g. [IS_RETRYABLE, IS_GRPC], then expose this with a method like Types() []Type. Since an error can contain multiple error types this would have to be a list instead of just a single enum value.
IsHTTP + AsHTTP: This essentially conveys the same information as HTTPStatus() (code int, isHTTP bool), so from a functionality standpoint it's not an issue. I would rename IsHTTP to HasHTTP though, since errors can contain multiple types of data. I'm not sure I see the benefit of retrieving this information across two methods, though, what's wrong with the boolean?

Before answering any of these questions, need to better understand how "correctly" (recommended, or whatever is the right term) chained consumers should error? For me to know the "right" design, I need to understand better the usage. Here are some examples, and we can debate on them what is the correct way (eventually document this). For these examples assume to use logs:

First Example

A -> B (drops logs with invalid trace identifiers) -> C -> D -> E (exports to Loki)

For a request with 10 logs:

B finds 2 "logs" that are not valid, for example "trace.id" is 34 hex characters.
E tries to export the logs and 2 out of the 8 remaining (assuming B dropped the 2 invalid logs) are reject with 400 http request.

Can you please tell me exactly which component should propagate/append errors and how?

Second Exmple

                       / -> E - > F
        / ->  C -> D -
A -> B -               \ -> G -> H
        \
         \ -> I -> J -> K

For a request with 10 logs:

F fails to send 4 logs with 429 http request.
H fails to send 2 logs with 400 http request.
D drops 2 logs because of invalid trace identifiers (trace.id too large).
K fails to send 1 logs with INVALID_ARGUMENT grpc status code.

Can you please tell me exactly which component should propagate/append errors and how?

djaglowski reviewed Dec 5, 2023

View reviewed changes

consumer/consumererror/README.md Show resolved Hide resolved

bogdandrutu reviewed Dec 6, 2023

View reviewed changes

consumer/consumererror/consumererror.go Outdated Show resolved Hide resolved

consumer/consumererror/consumererror.go Outdated Show resolved Hide resolved

consumer/consumererror/consumererror.go Outdated Show resolved Hide resolved

atoulme mentioned this pull request Dec 19, 2023

Stabilize module consumer #9046

Open

8 tasks

mx-psi added this to the `go.opentelemetry.io/collector/component` 1.0 milestone Dec 20, 2023

evan-bradley force-pushed the issue-7047 branch 2 times, most recently from 41e95d2 to d417f44 Compare January 8, 2024 20:40

evan-bradley mentioned this pull request Jan 10, 2024

OTLP PartialSuccess responses should not be interpreted as errors, items should count as "rejected" in pipeline metrics #9243

Open

jmacd mentioned this pull request Jan 10, 2024

Treat PartialSuccess as Success #9260

Merged

mx-psi modified the milestones: `go.opentelemetry.io/collector/component` 1.0, `go.opentelemetry.io/collector/consumer` 1.0 Jan 24, 2024

mx-psi reviewed Jan 29, 2024

View reviewed changes

consumer/consumererror/consumererror.go Outdated Show resolved Hide resolved

consumer/consumererror/consumererror.go Outdated Show resolved Hide resolved

consumer/consumererror/partial.go Outdated Show resolved Hide resolved

mx-psi added needed-for-1.0 release:required-for-ga Must be resolved before GA release and removed needed-for-1.0 labels Feb 7, 2024

github-actions bot added the Stale label Feb 22, 2024

evan-bradley force-pushed the issue-7047 branch from d417f44 to 2082dfe Compare March 6, 2024 04:32

evan-bradley removed the Stale label Mar 6, 2024

github-actions bot added the Stale label Mar 21, 2024

evan-bradley removed the Stale label Apr 2, 2024

evan-bradley commented Apr 11, 2024

View reviewed changes

consumer/consumererror/internal/statusconversion/conversion.go Show resolved Hide resolved

evan-bradley commented Apr 11, 2024

View reviewed changes

consumer/consumererror/statuserrors.go Outdated Show resolved Hide resolved

evan-bradley marked this pull request as ready for review April 11, 2024 19:31

evan-bradley requested a review from a team as a code owner April 11, 2024 19:31

evan-bradley requested a review from mx-psi April 11, 2024 19:31

TylerHelmuth reviewed Apr 11, 2024

View reviewed changes

bogdandrutu reviewed Jul 30, 2024

View reviewed changes

evan-bradley added 2 commits July 30, 2024 15:00

Address PR feedback

2aa9e86

Merge remote-tracking branch 'upstream/main' into issue-7047

8c05443

dmitryax reviewed Aug 5, 2024

View reviewed changes

evan-bradley and others added 3 commits August 5, 2024 16:13

Address PR feedback

c8c40ef

Update consumer/consumererror/README.md

c26e31a

Co-authored-by: Dmitrii Anoshin <anoshindx@gmail.com>

Address PR feedback

3aadc29

evan-bradley and others added 10 commits August 5, 2024 16:34

Merge remote-tracking branch 'upstream/main' into issue-7047

a0b8bfd

Fix interface implementation

6f85be2

go mod tidy

b55f661

Reorder imports

ef40c0d

make crosslink

dba92c6

Fix CI

dd69f4a

Polish and tests

7300bda

Update changelog

e953920

Minor cleanup

a3fb82d

Merge remote-tracking branch 'upstream/main' into issue-7047

12ffaba

evan-bradley requested review from bogdandrutu and dmitryax August 8, 2024 21:29

bogdandrutu reviewed Aug 9, 2024

View reviewed changes

evan-bradley and others added 2 commits August 9, 2024 08:05

Update consumer/consumererror/error.go

b3cbde5

Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>

Address PR feedback

1d1affc

		Retryable: Errors are retryable if re-submitting data to a sink may result
		in a successful submission.

[consumer] Allow annotating consumer errors with metadata #9041

Are you sure you want to change the base?

[consumer] Allow annotating consumer errors with metadata #9041

Conversation

evan-bradley commented Dec 5, 2023

jmacd commented Jan 10, 2024

jmacd commented Jan 10, 2024

github-actions bot commented Feb 22, 2024

github-actions bot commented Mar 21, 2024

codecov bot commented Apr 11, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-bradley Aug 9, 2024 • edited Loading

Choose a reason for hiding this comment

dmitryax left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-bradley commented Aug 5, 2024

evan-bradley commented Aug 8, 2024

bogdandrutu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bogdandrutu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

evan-bradley commented Aug 9, 2024 • edited Loading

bogdandrutu commented Aug 9, 2024

evan-bradley commented Aug 9, 2024

bogdandrutu commented Aug 11, 2024 • edited Loading

codecov bot commented Apr 11, 2024 •

edited

Loading

evan-bradley Aug 9, 2024 •

edited

Loading

evan-bradley commented Aug 9, 2024 •

edited

Loading

bogdandrutu commented Aug 11, 2024 •

edited

Loading