-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[exporterhelper] Add WithRequestQueue option to the exporter #8853
[exporterhelper] Add WithRequestQueue option to the exporter #8853
Conversation
f389f5f
to
728d706
Compare
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #8853 +/- ##
==========================================
- Coverage 90.40% 90.39% -0.01%
==========================================
Files 344 346 +2
Lines 18058 18103 +45
==========================================
+ Hits 16325 16365 +40
- Misses 1401 1405 +4
- Partials 332 333 +1 ☔ View full report in Codecov by Sentry. |
728d706
to
c25c88e
Compare
0399b37
to
82696bf
Compare
8657a33
to
6b26a1d
Compare
eaad286
to
f2d5aee
Compare
f2d5aee
to
8cd573d
Compare
ebc3091
to
38b688b
Compare
ed9dcb4
to
5cdf3c4
Compare
config/configqueue/queue.go
Outdated
// NewPersistentQueueFactory. | ||
// This API is at the early stage of development and may change without backward compatibility | ||
// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved. | ||
type PersistentQueueConfig struct { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Would be interested to see how people decide to use this vs QueueConfig in their Configs. When do I pick one or the other?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
QueueConfig
is used only for exporters that don't support persistent queue. And PersistentQueueConfig
for exporters that support both. Maybe we don't need to support memory-queue-only exporters. My idea was that some exporters don't want to provide serializers but still want to use custom requests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My idea was that some exporters don't want to provide serializers but still want to use custom requests.
We can have some default serializers using pdata maybe in that case? May complicate too much the implementation design...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That would require asking user for Request->pdata translator. Right now we ask for pdata->Request only
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Otherwise, we can apply the conversion after the queue. This can significantly complicate the implementation, as you mentioned.
exporter/exporterhelper/common.go
Outdated
// This option should be used with the new exporter helpers New[Traces|Metrics|Logs]RequestExporter. | ||
// This API is at the early stage of development and may change without backward compatibility | ||
// until https://github.com/open-telemetry/opentelemetry-collector/issues/8122 is resolved. | ||
func WithRequestQueue(cfg configqueue.QueueConfig, queueFactory QueueFactory) Option { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of having a new public QueueFactory
what about we change this to func WithRequestQueue(queue Queue, numConsumers int)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure how to pass QueueCreateSettings
then. Should we move it to the Start method?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
QueueCreateSettings
- What is in there that user doesn't have access to or you think is "duplicate"?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We would need to ask the user to pass at least component.DataType
and exporter.CreateSettings
to the persistent queue constructor in addition to the exporter helper itself
80af70a
to
9c52a84
Compare
This PR was marked stale due to lack of activity. It will be closed in 14 days. |
7ec2695
to
da40531
Compare
db8cbee
to
902e47a
Compare
The new configuration interface for the end users provides a new `queue_size_items` option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field, `queue_size,` which will later be deprecated through a longer transition process.
902e47a
to
a6279d9
Compare
Introduce a way to enable queue in the new exporter helper with a developer interface suggested in #8248 (comment).
The new configuration interface for the end users provides a new
queue_size_items
option to limit the queue by a number of spans, log records, or metric data points. The previous way to limit the queue by number of requests is preserved under the same field,queue_size,
which will later be deprecated through a longer transition process.Tracking issue: #8122