🎉 Source Github: Add MultipleTokenAuthenticator #5223

gaart · 2021-08-05T12:29:44Z

What

Multiple token support added to the CDK.
Solving #3250

How

Now we have MultipleTokenAuthenticator which receives a comma-separated list of tokens and rotates them during API calls.

vitaliizazmic

Could you please explain how token rotating is implemented while exceeded key limits?

gaart · 2021-08-10T06:27:26Z

@vitaliizazmic

Could you please explain how token rotating is implemented while exceeded key limits?

There's a backoff_time mechanism which adds timeouts when the limits exceeded, this PR doesn't change that behaviour but allows to use multiple tokens rotating them after each API call

vitaliizazmic · 2021-08-11T14:09:42Z

@gaart sounds good. Don't forget to run tests and format.

gaart · 2021-08-11T17:28:16Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1121018057
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1121018057

…-feature-rotate-tokens

…-feature-rotate-tokens # Conflicts: # airbyte-integrations/connectors/source-github/source_github/source.py

gaart · 2021-08-16T14:35:56Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1135844526
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1135844526

…ure-rotate-tokens # Conflicts: # airbyte-cdk/python/unit_tests/sources/streams/http/auth/test_auth.py

gaart · 2021-08-16T15:00:58Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1135926556
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1135926556

lukebussey · 2021-08-16T15:43:16Z

Why pass a comma separated list of tokens when you can just pass over a List?

…ure-rotate-tokens

sherifnada

one requested change but otherwise looks great

sherifnada · 2021-08-16T17:16:17Z

airbyte-cdk/python/airbyte_cdk/sources/streams/http/auth/token.py

+
+
+class MultipleTokenAuthenticator(HttpAuthenticator):
+    def __init__(self, tokens: str, auth_method: str = "Bearer", auth_header: str = "Authorization"):


this should take a list of strings rather than a comma separated string

good point, updated

sherifnada

Added some requested improvements, feel free to merge once they are addressed!

sherifnada · 2021-08-16T17:24:08Z

airbyte-cdk/python/CHANGELOG.md

+## 0.1.10
+Upd multiple token support: switch to list of tokens
+
+## 0.1.9


do we need two versions for the same release? can we just make it a single version?

Probably not. But both are released already

sherifnada · 2021-08-16T17:25:52Z

airbyte-cdk/python/airbyte_cdk/sources/streams/http/auth/token.py

@@ -36,3 +37,14 @@ def __init__(self, token: str, auth_method: str = "Bearer", auth_header: str = "

    def get_auth_header(self) -> Mapping[str, Any]:
        return {self.auth_header: f"{self.auth_method} {self._token}"}
+
+
+class MultipleTokenAuthenticator(HttpAuthenticator):


could you add a doc string explaining what this does e.g:

Uses the input list of tokens for authentication in a round-robin fashion. This allows load balancing quota consumption across multiple tokens.

airbyte-integrations/connectors/source-github/source_github/spec.json

sherifnada · 2021-08-16T17:30:04Z

airbyte-integrations/connectors/source-github/source_github/spec.json

@@ -9,7 +9,7 @@
    "properties": {
      "access_token": {
        "type": "string",
-        "description": "Log into Github and then generate a <a href=\"https://github.com/settings/tokens\"> personal access token</a>.",
+        "description": "Log into Github and then generate a <a href=\"https://github.com/settings/tokens\"> personal access token</a>. Separate multiple tokens with \",\"",


Ideally, type: string would be an array of strings too. However for backwards compatibility this is not great.

Could you create an issue to update this to a list of strings once #5396 is resolved?

sherifnada · 2021-08-16T17:31:11Z

airbyte-integrations/connectors/source-github/source_github/source.py

                ]
        if repositories:
            repositories_list += repositories

        return list(set(repositories_list))

+    @staticmethod
+    def _get_authenticator(token: str):
+        if TOKEN_SEPARATOR not in token:


do we need a special edge case for this? multiple auth should still work with one token right?

…ec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

gaart · 2021-08-17T13:51:59Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1139518093
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1139518093

…-feature-rotate-tokens # Conflicts: # airbyte-integrations/connectors/source-github/source_github/source.py

gaart · 2021-08-18T17:04:35Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1143998890
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1143998890

gaart · 2021-08-19T04:41:19Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1145675767
❌ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1145675767

…-feature-rotate-tokens

gaart · 2021-08-19T05:01:15Z

/test connector=connectors/source-github

🕑 connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1145715532
✅ connectors/source-github https://github.com/airbytehq/airbyte/actions/runs/1145715532

This reverts commit 51d406d.

gaart · 2021-08-19T05:59:53Z

/publish connector=connectors/source-github

Error: No ref found for: gaart/3250-feature-rotate-tokens

Add multiple token authenticator

86018f1

github-actions bot added area/connectors Connector related issues CDK Connector Development Kit labels Aug 5, 2021

gaart changed the title ~~Add multiple token authenticator~~ Python CDK, Source Github: Add multiple token authenticator Aug 5, 2021

gaart linked an issue Aug 5, 2021 that may be closed by this pull request

Source Github: Allow rotating between multiple API keys #3250

Closed

gaart requested review from keu, vitaliizazmic and vovavovavovavova August 5, 2021 12:52

vitaliizazmic reviewed Aug 6, 2021

View reviewed changes

jrhizor temporarily deployed to more-secrets August 11, 2021 17:30 Inactive

gaart added 6 commits August 11, 2021 23:24

Merge branch 'master' of github.com:airbytehq/airbyte into gaart/3250…

6a9beb8

…-feature-rotate-tokens

Add MultipleTokenAuthenticator

d8a7437

Upd docs

db9609c

Refactor

278e0af

Upd docs

a57965d

Merge branch 'master' of github.com:airbytehq/airbyte into gaart/3250…

66e7be7

…-feature-rotate-tokens # Conflicts: # airbyte-integrations/connectors/source-github/source_github/source.py

jrhizor temporarily deployed to more-secrets August 16, 2021 14:38 Inactive

Merge branch 'gaart/python-cdk-multi-token-auth' into gaart/3250-feat…

439301c

…ure-rotate-tokens # Conflicts: # airbyte-cdk/python/unit_tests/sources/streams/http/auth/test_auth.py

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 16, 2021

Fix merge typo

f5cc341

gaart requested review from sherifnada and marcosmarxm and removed request for vovavovavovavova and keu August 16, 2021 14:51

jrhizor temporarily deployed to more-secrets August 16, 2021 15:03 Inactive

gaart added 2 commits August 16, 2021 19:16

Upd multiple token support: switch to list of tokens

035f665

Merge branch 'gaart/python-cdk-multi-token-auth' into gaart/3250-feat…

b0c0683

…ure-rotate-tokens

sherifnada suggested changes Aug 16, 2021

View reviewed changes

Upd multiple token support: refactoring

9f50c4e

gaart requested a review from sherifnada August 16, 2021 17:22

sherifnada approved these changes Aug 16, 2021

View reviewed changes

gaart and others added 3 commits August 17, 2021 13:05

Update airbyte-integrations/connectors/source-github/source_github/sp…

d3e2235

…ec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Update airbyte-integrations/connectors/source-github/source_github/sp…

c215314

…ec.json Co-authored-by: Sherif A. Nada <snadalive@gmail.com>

Cleanup, switch to MultipleTokenAuthenticator completely

0a8979e

jrhizor temporarily deployed to more-secrets August 17, 2021 13:54 Inactive

gaart added 2 commits August 18, 2021 20:02

Upd changelog

626e615

Merge branch 'master' of github.com:airbytehq/airbyte into gaart/3250…

26fc9a5

…-feature-rotate-tokens # Conflicts: # airbyte-integrations/connectors/source-github/source_github/source.py

jrhizor temporarily deployed to more-secrets August 18, 2021 17:08 Inactive

jrhizor temporarily deployed to more-secrets August 19, 2021 04:44 Inactive

gaart changed the title ~~Python CDK, Source Github: Add multiple token authenticator~~ 🎉 Source Github: Add MultipleTokenAuthenticator Aug 19, 2021

Merge branch 'master' of github.com:airbytehq/airbyte into gaart/3250…

8e0aaf9

…-feature-rotate-tokens

jrhizor temporarily deployed to more-secrets August 19, 2021 05:03 Inactive

gaart merged commit 51d406d into master Aug 19, 2021

gaart deleted the gaart/3250-feature-rotate-tokens branch August 19, 2021 05:41

gaart added a commit that referenced this pull request Aug 19, 2021

Revert "🎉 Source Github: Add MultipleTokenAuthenticator (#5223)"

d0fec03

This reverts commit 51d406d.

igrankova added connectors/sources-api connectors/source/github labels Jan 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Github: Add MultipleTokenAuthenticator #5223

🎉 Source Github: Add MultipleTokenAuthenticator #5223

gaart commented Aug 5, 2021

vitaliizazmic left a comment •

edited

Loading

gaart commented Aug 10, 2021

vitaliizazmic commented Aug 11, 2021

gaart commented Aug 11, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 16, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 16, 2021 •

edited by github-actions bot

Loading

lukebussey commented Aug 16, 2021

sherifnada left a comment

sherifnada Aug 16, 2021

gaart Aug 16, 2021

sherifnada left a comment

sherifnada Aug 16, 2021

gaart Aug 17, 2021

sherifnada Aug 16, 2021

sherifnada Aug 16, 2021

sherifnada Aug 16, 2021

gaart commented Aug 17, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 18, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading



		class MultipleTokenAuthenticator(HttpAuthenticator):
		def __init__(self, tokens: str, auth_method: str = "Bearer", auth_header: str = "Authorization"):

🎉 Source Github: Add MultipleTokenAuthenticator #5223

🎉 Source Github: Add MultipleTokenAuthenticator #5223

Conversation

gaart commented Aug 5, 2021

What

How

vitaliizazmic left a comment • edited Loading

Choose a reason for hiding this comment

gaart commented Aug 10, 2021

vitaliizazmic commented Aug 11, 2021

gaart commented Aug 11, 2021 • edited by github-actions bot Loading

gaart commented Aug 16, 2021 • edited by github-actions bot Loading

gaart commented Aug 16, 2021 • edited by github-actions bot Loading

lukebussey commented Aug 16, 2021

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Aug 16, 2021

Choose a reason for hiding this comment

gaart Aug 16, 2021

Choose a reason for hiding this comment

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Aug 16, 2021

Choose a reason for hiding this comment

gaart Aug 17, 2021

Choose a reason for hiding this comment

sherifnada Aug 16, 2021

Choose a reason for hiding this comment

sherifnada Aug 16, 2021

Choose a reason for hiding this comment

sherifnada Aug 16, 2021

Choose a reason for hiding this comment

gaart commented Aug 17, 2021 • edited by github-actions bot Loading

gaart commented Aug 18, 2021 • edited by github-actions bot Loading

gaart commented Aug 19, 2021 • edited by github-actions bot Loading

gaart commented Aug 19, 2021 • edited by github-actions bot Loading

gaart commented Aug 19, 2021 • edited by github-actions bot Loading

vitaliizazmic left a comment •

edited

Loading

gaart commented Aug 11, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 16, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 16, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 17, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 18, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading

gaart commented Aug 19, 2021 •

edited by github-actions bot

Loading