-
Notifications
You must be signed in to change notification settings - Fork 4.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Source Github: break point added for workflows_runs stream #13926
Merged
Merged
Changes from 7 commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
a4e5b0c
break point added for workflows_runs stream
grubberr 6cbdae5
github.md updated
grubberr 2d9924a
github.md updated
grubberr 97319d3
break_point just before start_date on 32 days
grubberr 66b9afa
test_stream_workflow_runs_read_incremental added
grubberr a36416d
Merge branch 'master' into grubberr/13925-source-github
grubberr 801e555
improve comments
grubberr 41c7254
Merge branch 'master' into grubberr/13925-source-github
grubberr edb3bbf
add PR description
grubberr 587f4ae
auto-bump connector version
octavia-squidington-iii File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -90,7 +90,7 @@ This connector outputs the following incremental streams: | |
* [Review comments](https://docs.github.com/en/rest/reference/pulls#list-review-comments-in-a-repository) | ||
* [Reviews](https://docs.github.com/en/rest/reference/pulls#list-reviews-for-a-pull-request) | ||
* [Stargazers](https://docs.github.com/en/rest/reference/activity#list-stargazers) | ||
* [WorkflowRuns](https://docs.github.com/en/rest/reference/actions#list-workflow-runs-for-a-repository) | ||
* [WorkflowRuns](https://docs.github.com/en/rest/actions/workflow-runs#list-workflow-runs-for-a-repository) | ||
* [Workflows](https://docs.github.com/en/rest/reference/actions#workflows) | ||
|
||
### Notes | ||
|
@@ -99,12 +99,16 @@ This connector outputs the following incremental streams: | |
* read only new records; | ||
* output only new records. | ||
|
||
2. Other 20 incremental streams are also incremental but with one difference, they: | ||
2. Stream `workflow_runs` is almost pure incremental: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @Amruta-Ranade could you proofread these docs? |
||
* read new records and some portion of old records (in past 30 days) [docs](https://docs.github.com/en/actions/managing-workflow-runs/re-running-workflows-and-jobs); | ||
* output only new records. | ||
|
||
3. Other 19 incremental streams are also incremental but with one difference, they: | ||
* read all records; | ||
* output only new records. | ||
Please, consider this behaviour when using those 20 incremental streams because it may affect you API call limits. | ||
Please, consider this behaviour when using those 19 incremental streams because it may affect you API call limits. | ||
|
||
3. We are passing few parameters \(`since`, `sort` and `direction`\) to GitHub in order to filter records and sometimes for large streams specifying very distant `start_date` in the past may result in keep on getting error from GitHub instead of records \(respective `WARN` log message will be outputted\). In this case Specifying more recent `start_date` may help. | ||
4. We are passing few parameters \(`since`, `sort` and `direction`\) to GitHub in order to filter records and sometimes for large streams specifying very distant `start_date` in the past may result in keep on getting error from GitHub instead of records \(respective `WARN` log message will be outputted\). In this case Specifying more recent `start_date` may help. | ||
**The "Start date" configuration option does not apply to the streams below, because the GitHub API does not include dates which can be used for filtering:** | ||
|
||
* `assignees` | ||
|
@@ -137,6 +141,7 @@ The GitHub connector should not run into GitHub API limitations under normal usa | |
|
||
| Version | Date | Pull Request | Subject | | ||
|:--------|:-----------| :--- |:-------------------------------------------------------------------------------------------------------------| | ||
| 0.2.36 | 2022-06-20 | [13926](https://github.com/airbytehq/airbyte/pull/13926) | Break point added for `workflows_runs` stream | | ||
| 0.2.35 | 2022-06-16 | [13763](https://github.com/airbytehq/airbyte/pull/13763) | Use GraphQL for `pull_request_stats` stream | | ||
| 0.2.34 | 2022-06-14 | [13707](https://github.com/airbytehq/airbyte/pull/13707) | Fix API sorting, fix `get_starting_point` caching | | ||
| 0.2.33 | 2022-06-08 | [13558](https://github.com/airbytehq/airbyte/pull/13558) | Enable caching only for parent streams | | ||
|
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why 32 and not 30?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I have played with this 30 days border edge and I found a couple of things:
Objects are "almost" sorted - from time to time some near objects are out of order. I think it's internal details of the GitHub messenging queue.
But it's no big deal because it's related to pretty close objects.
I have also played with object mutation on this 30 days border and it looks like this border is blurred a little bit.
I just decided to add a couple of days to get a 100% guarantee that the object will not mutate anymore.