-
Notifications
You must be signed in to change notification settings - Fork 821
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(otlp-exporter-base): add retries #3207
feat(otlp-exporter-base): add retries #3207
Conversation
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
For some reason, using
These events should fire depending on when the request is aborted (before socket assigned, before connection succeeds or after response is received). If I use Any idea what I'm missing here or what's going on? If you run this test and use |
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## main #3207 +/- ##
==========================================
- Coverage 93.79% 93.62% -0.18%
==========================================
Files 264 265 +1
Lines 7349 7417 +68
Branches 1491 1511 +20
==========================================
+ Hits 6893 6944 +51
- Misses 456 473 +17
|
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Those events only fire if you call |
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
I would love some feedback on the current implementation of the retry logic. I still need to confirm and add the following items:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overall LGTM, thank you for working on this!
experimental/packages/otlp-exporter-base/src/platform/browser/util.ts
Outdated
Show resolved
Hide resolved
experimental/packages/otlp-exporter-base/src/platform/node/util.ts
Outdated
Show resolved
Hide resolved
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
…/opentelemetry-js into add-retries-to-exporters
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
…/opentelemetry-js into add-retries-to-exporters
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
I've added the following:
|
I'm unable to add real node tests to test this retry logic because the tests are very flaky when I use a real node http request. I tried to use "nock" but I found out that it overrides the client http request so I can't test the retry logic. |
I am wondering if you could use |
experimental/packages/otlp-exporter-base/src/platform/browser/util.ts
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay.
I'm wondering if it would be possible to unify the retry handling around the sendWithXhr
and sendWithHttp
-- it only depends on the 'Retry-After'
header and the response status code.
It may be simpler if the platform-dependent part can be minimized.
Sure I can give that a shot after thanksgiving break. |
@legendecas Regarding your comment:
Sorry for the delay on this. I don't think I can unify the retry handling between |
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
I think we should create a dedicated http client abstraction on both Node.js and browser, either by using widely adopted npm package (like superagent or axios) or creating our own one (since we may need to tweak it to be able to use With the only comment left (#3207 (comment)) resolved, I think this should be ready to land. |
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
…/opentelemetry-js into add-retries-to-exporters
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
…/opentelemetry-js into add-retries-to-exporters
Signed-off-by: Svetlana Brennan <50715937+svetlanabrennan@users.noreply.github.com>
I've updated the |
I'm going to merge this as @opentelemetry/instrumentation-http:test failure on node-windows-tests is being tracked at #3628. |
Which problem is this PR solving?
Adding retries to otlp http & proto exporters.
Note: this pr does not add retries to the grpc otlp exporter.
Fixes #1233
Short description of the changes
Added retry logic to
sendWithHttp
andsendWithXhr
functions. It does the following:If the exporter gets a failed response that is retryable, we check if the response included a "retry-after" header.
This logic is retried until there is a successful response or until the exporter timer times out (set via
timeoutMillis
in the exporter config or via env vars). If the exporter timers times outs, the request is aborted with a "Request Timeout" error.Type of change
Please delete options that are not relevant.
How Has This Been Tested?
Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration
Checklist: