-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 New Source: Monday #7168
🎉 New Source: Monday #7168
Conversation
Hey @tuanchris thanks for the contribution. We will review it soon. Can you share the credentials over slack (DM) for me test it out. |
Please share unit test and integration tests screen shoot showing they are passing locally. |
@marcosmarxm Here are the tests: Integration test:
I tried looking at the two output file: |
…marcos/test-pr-7168
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please see comments
def should_retry(self, response: requests.Response) -> bool: | ||
# Monday API return code 200 with and errors key if complexity is too high. | ||
# https://api.developer.monday.com/docs/complexity-queries | ||
is_complex_query = response.json().get("errors") | ||
return response.status_code == 429 or 500 <= response.status_code < 600 or is_complex_query |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This will request the same query? If its a complex query this means will keep in a loop?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I ran integration tests and got backoff error, I need to add a logging to understand the error:
2021-10-26 01:10:13 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - [{'message': "Field 'users' doesn't accept argument 'page'", 'locations': [{'line': 1, 'column': 16}], 'fields': ['query', 'users', 'page']}]
2021-10-26 01:10:13 INFO () DefaultAirbyteStreamFactory(internalLog):86 - Backing off _send(...) for 15.0s (airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException)
2021-10-26 01:10:29 INFO () DefaultAirbyteStreamFactory(internalLog):86 - Caught retryable error '' after 1 tries. Waiting 15 seconds then retrying...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
According to this document, the free trial account (which we are on) can only call queries with 1 mil or less per minute. In case of integration test, I think we reached the limit and this is the response.txt
from the request:
{"errors":[{"message":"Complexity budget exhausted, query cost 202535 budget remaining 189860 out of 1000000 reset in 43 seconds"}],"account_id":10204492}
The code above is meant to check if status_code == 200
and there is an errors
key in the response.json()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Got it, the question is: if a query can stay in Complex Query and always keep in backoff.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think this is possible. From the same link: Complexity levels are calculated based on the query.
. If a query return more than the complexity limit, we couldn't do the initial sync. The limit was hit because, with the integration test, we are running many successive queries in a short period of time. The retry and backoff will handle that.
…urce.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @tuanchris
* add items stream * add boards stream * add remaining streams * check point * lint * fix unit test & linting * Delete launch.json * fix schema * add retry logic * Update airbyte-integrations/connectors/source-monday/source_monday/source.py Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> * small fix + add creds * bump version and config file Co-authored-by: Marcos Marx <marcosmarxm@gmail.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com>
What
How
Describe the solution
Recommended reading order
x.java
y.python
Pre-merge Checklist
Expand the relevant checklist and delete the others.
New Connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/SUMMARY.md
docs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampledocs/integrations/README.md
airbyte-integrations/builds.md
Airbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing./publish
command described hereUpdating a connector
Community member or Airbyter
airbyte_secret
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
.README.md
bootstrap.md
. See description and examplesdocs/integrations/<source or destination>/<name>.md
including changelog. See changelog exampleAirbyter
If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.
/test connector=connectors/<name>
command is passing./publish
command described hereConnector Generator
-scaffold
in their name) have been updated with the latest scaffold by running./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates
then checking in your changes