-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HTTPOperator should have the possibility to retry at the page level using the http hook run_with_advanced_retry #39247
Labels
Comments
lopezvit
added
kind:feature
Feature Requests
needs-triage
label for new issues that we didn't triage yet
labels
Apr 25, 2024
potiuk
added
good first issue
and removed
needs-triage
label for new issues that we didn't triage yet
labels
Apr 25, 2024
I have created this extended version and seems to be working correctly, sorry I don't have time to create a PR:
|
Hi, I can work on this issue. Could you please assign me? |
Sorry, I'm not a collaborator, so I cannot assign anybody... maybe @eladkal could? |
96 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
I should be possible that the HTTPOperator retries some of the request without failing the whole task, since when it is using pagination, some pages might fail.
Use case/motivation
I'm requesting all the pages from an API endpoint (TeamTailor JobPosts) and that is a total of 438 pages.
Sometimes while going through these pages the server returns error 500.
If that page in particular would be retried with exponential backoff the whole task would probably succeed, but retrying the whole task might reproduce the problem.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: