🎉 Source github: add oauth spec file #6761

avida · 2021-10-05T09:23:16Z

What

Resolves #6263

How

Describe the solution

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Updating a connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
Credentials added to Github CI. Instructions.
/test connector=connectors/<name> command is passing.
New Connector version released on Dockerhub by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed.

avida · 2021-10-05T10:15:42Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307379327
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307379327
🐛 https://gradle.com/s/abdixb2nrejlm

htrueman

Please change the PR title.
Also I think the SAT update should be placed in separate PR.

htrueman · 2021-10-05T10:16:54Z

...integrations/bases/source-acceptance-test/source_acceptance_test/utils/json_schema_helper.py

@@ -164,6 +164,22 @@ def get_expected_schema_structure(schema: dict, annotate_one_of: bool = False) -
    :returns list of object property keys paths
    """
    paths = []
+    if "$ref" in schema:


hm, I'm not sure we should update SAT in this PR

otherwise it wouldnt work with github schemas

avida · 2021-10-05T10:26:16Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307412962
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307412962
🐛 https://gradle.com/s/3i66xieazjiww

avida · 2021-10-05T10:39:19Z

/publish connector=bases/source-acceptance-test

🕑 bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1307455658
✅ bases/source-acceptance-test https://github.com/airbytehq/airbyte/actions/runs/1307455658

avida · 2021-10-05T10:45:47Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307475935
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1307475935
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                        Stmts   Miss  Cover
	 -----------------------------------------------
	 source_github/__init__.py       2      0   100%
	 source_github/source.py        72     48    33%
	 source_github/streams.py      367    210    43%
	 -----------------------------------------------
	 TOTAL                         441    258    41%

sherifnada · 2021-10-05T15:55:34Z

airbyte-integrations/connectors/source-github/source_github/source.py

@@ -64,7 +64,11 @@ def _generate_repositories(config: Mapping[str, Any], authenticator: MultipleTok
        return list(set(repositories_list))

    @staticmethod
-    def _get_authenticator(token: str):
+    def _get_authenticator(config: Dict[str, Any]):
+        token = config.get("access_token")


can you add a comment mentioning this is for backwards compatibility?

Suggested change

token = config.get("access_token")

# Before we supported oauth, personal_access_token was called `access_token` and it lived at the

# config root. So we first check to make sure any backwards compatbility is handled.

token = config.get("access_token")

sherifnada · 2021-10-05T15:56:18Z

airbyte-integrations/connectors/source-github/source_github/spec.json

-        "airbyte_secret": true
+      "credentials": {
+        "title": "Authentication mechanism",
+        "description": "Choose either to provide the API key or the OAuth2.0 credentials",


not having a specific reference makes it less likely to go out of date ;)

Suggested change

"description": "Choose either to provide the API key or the OAuth2.0 credentials",

"description": "Choose how to authenticate to Github",

sherifnada · 2021-10-05T15:56:51Z

airbyte-integrations/connectors/source-github/source_github/spec.json

+              },
+              "access_token": {
+                "type": "string",
+                "title": "Access Tokens",


Suggested change

"title": "Access Tokens",

"title": "Access Token",

sherifnada · 2021-10-05T15:57:37Z

airbyte-integrations/connectors/source-github/source_github/spec.json

        "description": "The date from which you'd like to replicate data for GitHub in the format YYYY-MM-DDT00:00:00Z. All data generated after this date will be replicated. Note that it will be used only in the following incremental streams: comments, commits and issues.",
        "examples": ["2021-03-01T00:00:00Z"],
        "pattern": "^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}Z$"
      },
      "branch": {
        "type": "string",
+        "title": "Branch",


thanks for adding titles for other options as well! Users will be very thankful ;)

avida · 2021-10-05T18:50:48Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1308924372
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1308924372

avida · 2021-10-05T19:21:13Z

/publish connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1309029525
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1309029525

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 5, 2021

avida temporarily deployed to more-secrets October 5, 2021 09:24 Inactive

Source github: add oauth spec file

e687cc2

avida force-pushed the drezchykov/github-oauth branch from e48ba0c to e687cc2 Compare October 5, 2021 09:24

avida temporarily deployed to more-secrets October 5, 2021 09:26 Inactive

avida requested review from Zirochkaa, yevhenii-ldv and htrueman October 5, 2021 09:42

jrhizor temporarily deployed to more-secrets October 5, 2021 10:18 Inactive

htrueman reviewed Oct 5, 2021

View reviewed changes

avida changed the title ~~Source github: add oauth spec file~~ 🎉 Source github: add oauth spec file Oct 5, 2021

jrhizor temporarily deployed to more-secrets October 5, 2021 10:28 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 10:41 Inactive

htrueman approved these changes Oct 5, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets October 5, 2021 10:48 Inactive

avida requested a review from sherifnada October 5, 2021 10:52

sherifnada approved these changes Oct 5, 2021

View reviewed changes

fix review comments

16b93d1

avida temporarily deployed to more-secrets October 5, 2021 18:51 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 18:53 Inactive

jrhizor temporarily deployed to more-secrets October 5, 2021 19:23 Inactive

avida merged commit 78fd9d3 into master Oct 5, 2021

avida deleted the drezchykov/github-oauth branch October 5, 2021 19:28

jrhizor mentioned this pull request Oct 5, 2021

Bump Airbyte version from 0.30.14-alpha to 0.30.15-alpha #6808

Merged

schlattk pushed a commit to schlattk/airbyte that referenced this pull request Jan 4, 2022

🎉 Source github: add oauth spec file (airbytehq#6761)

1f83555

igrankova added connectors/sources-api connectors/source/github labels Jan 6, 2022

alexkreidler mentioned this pull request May 22, 2023

[EPIC] Support oauth via the UI in OSS #13021

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source github: add oauth spec file #6761

🎉 Source github: add oauth spec file #6761

avida commented Oct 5, 2021

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

htrueman left a comment

htrueman Oct 5, 2021

avida Oct 5, 2021

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

sherifnada Oct 5, 2021

sherifnada Oct 5, 2021

sherifnada Oct 5, 2021

sherifnada Oct 5, 2021

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

	"description": "Choose either to provide the API key or the OAuth2.0 credentials",
	"description": "Choose how to authenticate to Github",

🎉 Source github: add oauth spec file #6761

🎉 Source github: add oauth spec file #6761

Conversation

avida commented Oct 5, 2021

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

avida commented Oct 5, 2021 • edited by github-actions bot Loading

htrueman left a comment

Choose a reason for hiding this comment

htrueman Oct 5, 2021

Choose a reason for hiding this comment

avida Oct 5, 2021

Choose a reason for hiding this comment

avida commented Oct 5, 2021 • edited by github-actions bot Loading

avida commented Oct 5, 2021 • edited by github-actions bot Loading

avida commented Oct 5, 2021 • edited by github-actions bot Loading

sherifnada Oct 5, 2021

Choose a reason for hiding this comment

sherifnada Oct 5, 2021

Choose a reason for hiding this comment

sherifnada Oct 5, 2021

Choose a reason for hiding this comment

sherifnada Oct 5, 2021

Choose a reason for hiding this comment

avida commented Oct 5, 2021 • edited by github-actions bot Loading

avida commented Oct 5, 2021 • edited by github-actions bot Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading

avida commented Oct 5, 2021 •

edited by github-actions bot

Loading