-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Resolving issue #1353: implement backoff in Google Sheets #1438
Resolving issue #1353: implement backoff in Google Sheets #1438
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@yevhenii-ldv This looks great so far, but we aren't backing off on every client call. For example in the discovery
method we make a call using client
but there is no backing off there as far as i can tell. Could we add this backoff to every call made with the client? It may make sense to put the client in a separate class if it helps.
@sherifnada PR was updated with new client`s class for Google Sheets requests with backoff |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merry christmas @yevhenii-ldv! looking much better already. Two more things needed:
- Update the version of this connector in the Dockerfile, YAML, and generated JSON file. We generally want to do this whenever we make an update to a connector so we can release a new version.
- small structural change to the client
airbyte-integrations/connectors/source-google-sheets/google_sheets_source/client.py
Show resolved
Hide resolved
@sherifnada I have updated the PR as per your comments. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good!
I'm going to build the image, run the tests, and then merge this.
What
Using a backoff in the Google Sheets source code when the request limit is reached.
How
Using python library
backoff
Pre-merge Checklist
Recommended reading order
test.java
component.ts