[exporterhelper] Add WithRequestQueue option to the exporter #8853

dmitryax · 2023-11-12T17:03:36Z

Introduce a way to enable queue in the new exporter helper with a developer interface suggested in #8248 (comment).

The new configuration interface for the end users provides a new queue_size_items option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, queue_size, which will later be deprecated through a longer transition process.

Tracking issue: #8122

codecov · 2023-11-12T18:15:43Z

Codecov Report

Attention: 3 lines in your changes are missing coverage. Please review.

Comparison is base (7abb962) 90.40% compared to head (a6279d9) 90.39%.

Files	Patch %	Lines
exporter/exporterhelper/logs.go	66.66%	0 Missing and 1 partial ⚠️
exporter/exporterhelper/metrics.go	66.66%	0 Missing and 1 partial ⚠️
exporter/exporterhelper/traces.go	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8853      +/-   ##
==========================================
- Coverage   90.40%   90.39%   -0.01%     
==========================================
  Files         344      346       +2     
  Lines       18058    18103      +45     
==========================================
+ Hits        16325    16365      +40     
- Misses       1401     1405       +4     
- Partials      332      333       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

exporter/exporterhelper/internal/request/request.go

exporter/exporterhelper/internal/queue_capacity.go

bogdandrutu · 2023-12-29T18:11:21Z

config/configqueue/queue.go

+// NewPersistentQueueFactory.
+// This API is at the early stage of development and may change without backward compatibility
+// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.
+type PersistentQueueConfig struct {


Would be interested to see how people decide to use this vs QueueConfig in their Configs. When do I pick one or the other?

QueueConfig is used only for exporters that don't support persistent queue. And PersistentQueueConfig for exporters that support both. Maybe we don't need to support memory-queue-only exporters. My idea was that some exporters don't want to provide serializers but still want to use custom requests.

My idea was that some exporters don't want to provide serializers but still want to use custom requests.

We can have some default serializers using pdata maybe in that case? May complicate too much the implementation design...

That would require asking user for Request->pdata translator. Right now we ask for pdata->Request only

Otherwise, we can apply the conversion after the queue. This can significantly complicate the implementation, as you mentioned.

config/configqueue/queue.go

consumer/go.mod

exporter/debugexporter/go.mod

exporter/exporterhelper/internal/bounded_memory_queue.go

exporter/exporterhelper/internal/persistent_queue.go

bogdandrutu · 2023-12-29T18:22:00Z

exporter/exporterhelper/common.go

+// This option should be used with the new exporter helpers New[Traces|Metrics|Logs]RequestExporter.
+// This API is at the early stage of development and may change without backward compatibility
+// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved.
+func WithRequestQueue(cfg configqueue.QueueConfig, queueFactory QueueFactory) Option {


Instead of having a new public QueueFactory what about we change this to func WithRequestQueue(queue Queue, numConsumers int)

Not sure how to pass QueueCreateSettings then. Should we move it to the Start method?

QueueCreateSettings - What is in there that user doesn't have access to or you think is "duplicate"?

We would need to ask the user to pass at least component.DataType and exporter.CreateSettings to the persistent queue constructor in addition to the exporter helper itself

…9197) Address #8853 (comment)

github-actions · 2024-01-18T03:15:46Z

This PR was marked stale due to lack of activity. It will be closed in 14 days.

exporter/exporterqueue/queue.go

exporter/internal/err.go

exporter/exporterqueue/queue.go

The new configuration interface for the end users provides a new `queue_size_items` option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, `queue_size,` which will later be deprecated through a longer transition process.

dmitryax requested a review from a team as a code owner November 12, 2023 17:03

dmitryax requested a review from Aneurysm9 November 12, 2023 17:03

dmitryax force-pushed the request-exporter-queue branch 3 times, most recently from f389f5f to 728d706 Compare November 12, 2023 18:12

dmitryax commented Nov 12, 2023

View reviewed changes

exporter/exporterhelper/internal/request/request.go Outdated Show resolved Hide resolved

dmitryax force-pushed the request-exporter-queue branch from 728d706 to c25c88e Compare November 18, 2023 00:01

dmitryax marked this pull request as draft November 19, 2023 18:34

dmitryax force-pushed the request-exporter-queue branch 2 times, most recently from 0399b37 to 82696bf Compare November 19, 2023 18:40

dmitryax force-pushed the request-exporter-queue branch 7 times, most recently from 8657a33 to 6b26a1d Compare December 1, 2023 01:07

dmitryax force-pushed the request-exporter-queue branch 3 times, most recently from eaad286 to f2d5aee Compare December 10, 2023 00:09

dmitryax marked this pull request as ready for review December 10, 2023 00:09

dmitryax force-pushed the request-exporter-queue branch from f2d5aee to 8cd573d Compare December 10, 2023 01:58

dmitryax mentioned this pull request Dec 12, 2023

Deprecate exporterhelper.RetrySettings in favor of configretry.BackOffConfig #9091

Merged

splunkericl reviewed Dec 13, 2023

View reviewed changes

exporter/exporterhelper/internal/queue_capacity.go Outdated Show resolved Hide resolved

dmitryax force-pushed the request-exporter-queue branch 5 times, most recently from ebc3091 to 38b688b Compare December 19, 2023 19:03

dmitryax force-pushed the request-exporter-queue branch 5 times, most recently from ed9dcb4 to 5cdf3c4 Compare December 27, 2023 06:20

bogdandrutu reviewed Dec 29, 2023

View reviewed changes

dmitryax mentioned this pull request Dec 31, 2023

[chore] [exporterhelper] Move queue creation parameters to structs #9197

Merged

dmitryax added a commit that referenced this pull request Jan 2, 2024

[chore] [exporterhelper] Move queue creation parameters to structs (#…

7f7b826

…9197) Address #8853 (comment)

dmitryax force-pushed the request-exporter-queue branch 2 times, most recently from 80af70a to 9c52a84 Compare January 3, 2024 16:35

github-actions bot added the Stale label Jan 18, 2024

dmitryax removed the Stale label Jan 18, 2024

dmitryax mentioned this pull request Jan 25, 2024

[exporterhelper] Deprecate QueueSettings, use QueueConfig instead #9393

Closed

dmitryax force-pushed the request-exporter-queue branch 3 times, most recently from 7ec2695 to da40531 Compare January 29, 2024 23:25

bogdandrutu reviewed Jan 30, 2024

View reviewed changes

exporter/exporterqueue/queue.go Outdated Show resolved Hide resolved

exporter/internal/err.go Outdated Show resolved Hide resolved

bogdandrutu reviewed Jan 30, 2024

View reviewed changes

exporter/exporterqueue/queue.go Outdated Show resolved Hide resolved

exporter/exporterqueue/queue.go Outdated Show resolved Hide resolved

dmitryax force-pushed the request-exporter-queue branch 4 times, most recently from db8cbee to 902e47a Compare February 2, 2024 22:27

dmitryax force-pushed the request-exporter-queue branch from 902e47a to a6279d9 Compare February 2, 2024 22:44

bogdandrutu approved these changes Feb 3, 2024

View reviewed changes

bogdandrutu merged commit bdbb5f3 into open-telemetry:main Feb 3, 2024
46 checks passed

github-actions bot added this to the next release milestone Feb 3, 2024

dmitryax deleted the request-exporter-queue branch February 5, 2024 02:13

dmitryax mentioned this pull request Feb 5, 2024

[exporterhelper] User interface for controling queue measurement #9462

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporterhelper] Add WithRequestQueue option to the exporter #8853

[exporterhelper] Add WithRequestQueue option to the exporter #8853

dmitryax commented Nov 12, 2023 •

edited

Loading

codecov bot commented Nov 12, 2023 •

edited

Loading

bogdandrutu Dec 29, 2023

dmitryax Dec 29, 2023

bogdandrutu Dec 30, 2023

dmitryax Dec 31, 2023

dmitryax Jan 29, 2024

bogdandrutu Dec 29, 2023

dmitryax Dec 29, 2023

bogdandrutu Dec 30, 2023

dmitryax Dec 31, 2023 •

edited

Loading

github-actions bot commented Jan 18, 2024

[exporterhelper] Add WithRequestQueue option to the exporter #8853

[exporterhelper] Add WithRequestQueue option to the exporter #8853

Conversation

dmitryax commented Nov 12, 2023 • edited Loading

codecov bot commented Nov 12, 2023 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dmitryax Dec 31, 2023 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jan 18, 2024

dmitryax commented Nov 12, 2023 •

edited

Loading

codecov bot commented Nov 12, 2023 •

edited

Loading

dmitryax Dec 31, 2023 •

edited

Loading