Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Source Github: add reaction streams #5860

Merged
merged 7 commits into from
Sep 9, 2021
Merged

🎉 Source Github: add reaction streams #5860

merged 7 commits into from
Sep 9, 2021

Conversation

yevhenii-ldv
Copy link
Contributor

@yevhenii-ldv yevhenii-ldv commented Sep 6, 2021 •
edited
Loading

What

Closes #5499.

How

Describe the solution

Recommended reading order

  1. x.java
  2. y.python

Pre-merge Checklist

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

  • Create a non-forked branch based on this PR and test the below items on it
  • Build is successful
  • Credentials added to Github CI. Instructions.
  • /test connector=connectors/<name> command is passing.
  • New Connector version released on Dockerhub by running the /publish command described here

@github-actions github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 6, 2021
@yevhenii-ldv
Copy link
Contributor Author

yevhenii-ldv commented Sep 6, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1206253363
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1206253363

@jrhizor jrhizor temporarily deployed to more-secrets September 6, 2021 14:01 Inactive
@yevhenii-ldv yevhenii-ldv changed the title Source Github: add reaction streams 🎉 Source Github: add reaction streams Sep 6, 2021
htrueman
htrueman approved these changes Sep 6, 2021
View reviewed changes
Copy link
Contributor

@htrueman htrueman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Added a few questions.

airbyte-integrations/connectors/source-github/source_github/streams.py Outdated Show resolved Hide resolved
...te-integrations/connectors/source-github/source_github/schemas/commit_comment_reactions.json Show resolved Hide resolved
@yevhenii-ldv yevhenii-ldv removed this from the Connectors, September 17th 2021 milestone Sep 7, 2021
sherifnada
sherifnada approved these changes Sep 7, 2021
View reviewed changes
airbyte-integrations/connectors/source-github/source_github/streams.py

def stream_slices(self, **kwargs) -> Iterable[Optional[Mapping[str, Any]]]:
for stream_slice in super().stream_slices(**kwargs):
for parent_record in self._parent_stream.read_records(sync_mode=SyncMode.full_refresh, stream_slice=stream_slice):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not for this PR, but it does feel like we should cache these. Lots of parent streams going on atm

sherifnada
sherifnada reviewed Sep 7, 2021
View reviewed changes
airbyte-integrations/connectors/source-github/source_github/streams.py Outdated
class ReactionStream(GithubStream, ABC):

parent_key = "id"
fields_to_minimize = ("user",)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we shouldn't minimize this since it may contain new users not in any other streams

Suggested change
fields_to_minimize = ("user",)

sherifnada
sherifnada reviewed Sep 7, 2021
View reviewed changes
airbyte-integrations/connectors/source-github/source_github/streams.py

def __init__(self, **kwargs):
self._parent_stream = self.parent_entity(**kwargs)
kwargs.pop("start_date", None)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we make a copy of kwargs in case it is reused across streams?

@yevhenii-ldv
Copy link
Contributor Author

yevhenii-ldv commented Sep 8, 2021 •
edited by github-actions bot
Loading

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1214994295
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1214994295

@jrhizor jrhizor temporarily deployed to more-secrets September 8, 2021 21:28 Inactive
@yevhenii-ldv
Copy link
Contributor Author

yevhenii-ldv commented Sep 9, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1216850988
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1216850988

@jrhizor jrhizor temporarily deployed to more-secrets September 9, 2021 10:08 Inactive
@yevhenii-ldv yevhenii-ldv merged commit 9b71c28 into master Sep 9, 2021
@yevhenii-ldv yevhenii-ldv deleted the ykruochkin/add-reactions-streams branch September 9, 2021 10:13
@etsybaev
Copy link
Contributor

etsybaev commented Sep 9, 2021 •
edited
Loading

Hi there. It seems like after this PR (at least this is the first where we see it) we started getting some failed checks on CI.
Could it be caused by these changes?
For example Acceptance Tests (Kube) - Run Kubernates End to End tests and Platform Build - Automatic Migration Acceptance tests.
Thanks!
Selection_076

@yevhenii-ldv
Copy link
Contributor Author

Hi @etsybaev,
I think that this PR could not break the SI.

  1. Airbyte CI / Ensure all required Docker images exist on Dockerhub (push) - test failed because I updated the version of the connector before publishing it.
  2. Airbyte CI / Connectors Base: Build (push) and Airbyte CI / Platform: Build (push) - we can see next logs:
Run git --no-pager diff && test -z "$(git --no-pager diff)"
##[debug]/usr/bin/bash -e /actions-runner/_work/_temp/964be428-95ff-4c75-9d5f-d8fc2ac216a6.sh
diff --git a/airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java b/airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java
index 8acb9c8..b3fa7aa 100644
--- a/airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java
+++ b/airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java

error with airbyte-db/lib/src/main/java/io/airbyte/db/mongodb/MongoUtils.java file, this file has not been changed in this PR.

@jrhizor jrhizor mentioned this pull request Sep 16, 2021
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sherifnada sherifnada sherifnada approved these changes

@Zirochkaa Zirochkaa Zirochkaa approved these changes

@htrueman htrueman htrueman approved these changes

@avida avida Awaiting requested review from avida

@davinchia davinchia Awaiting requested review from davinchia

@tuliren tuliren Awaiting requested review from tuliren

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/github connectors/sources-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Source github - Add reactions as a stream
8 participants
@yevhenii-ldv @etsybaev @sherifnada @htrueman @Zirochkaa @jrhizor @VasylLazebnyk @igrankova