Source Pinterest: support OAuth #16271

arsenlosenko · 2022-09-02T13:33:09Z

What

Resolving:
#7754

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

CLAassistant · 2022-09-02T13:33:16Z

All committers have signed the CLA.

arsenlosenko · 2022-09-02T14:28:43Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2979793907
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2979793907
🐛 https://gradle.com/s/zqihrcz3l7oi4

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - source_...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_core.py::TestConnection::test_check[inputs2] - FileNotFoundError: ...
ERROR test_core.py::TestDiscovery::test_discover[inputs1] - FileNotFoundError...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs1] - Fil...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs1]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - Fil...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
======== 3 failed, 25 passed, 1 skipped, 10 errors in 422.36s (0:07:02) ========

airbyte-integrations/connectors/source-pinterest/source_pinterest/source.py

docs/integrations/sources/pinterest.md

arsenlosenko · 2022-09-03T11:17:38Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2984193280
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2984193280
🐛 https://gradle.com/s/tzxarnb6wcx4m

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_two_sequential_reads[inputs0]
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_core.py::TestConnection::test_check[inputs2] - FileNotFoundError: ...
ERROR test_core.py::TestDiscovery::test_discover[inputs1] - FileNotFoundError...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs1] - Fil...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs1]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - Fil...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:57: Backward compatibility tests are disabled for version 0.1.2.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
======== 6 failed, 21 passed, 2 skipped, 10 errors in 647.62s (0:10:47) ========

arsenlosenko · 2022-09-04T06:35:07Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2987107916
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2987107916
🐛 https://gradle.com/s/27ezgpjd6b5gw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - so...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
FAILED test_incremental.py::TestIncremental::test_read_sequential_slices[inputs0]
ERROR test_core.py::TestConnection::test_check[inputs2] - FileNotFoundError: ...
ERROR test_core.py::TestDiscovery::test_discover[inputs1] - FileNotFoundError...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs1-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs1]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs1] - Fil...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs1]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs1] - Fil...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:57: Backward compatibility tests are disabled for version 0.1.2.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
======== 3 failed, 24 passed, 2 skipped, 10 errors in 395.87s (0:06:35) ========

…ource-pinterest-support-oauth-cp

bazarnov · 2022-09-05T09:14:58Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2992069604
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2992069604
🐛 https://gradle.com/s/teg2gr2bvccza

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - Failed: Please check...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:57: Backward compatibility tests are disabled for version 0.1.2.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.2.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:23: `future_state_path` not specified, skipping
============= 1 failed, 34 passed, 4 skipped in 272.33s (0:04:32) ==============

…duce API Calls for child streams, commented out Incremental and Full refresh SAT tests

…ource-pinterest-support-oauth-cp

bazarnov · 2022-09-05T09:46:09Z

/test connector=connectors/source-pinterest

🕑 connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2992254066
❌ connectors/source-pinterest https://github.com/airbytehq/airbyte/actions/runs/2992254066
🐛 https://gradle.com/s/z7prtk55trynw

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestConnection::test_check[inputs2] - AssertionError: as...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: At l...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestFullRefresh.test_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:60: Skipping TestIncremental.test_two_sequential_reads because not found in the config
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:57: Backward compatibility tests are disabled for version 0.1.2.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:235: Backward compatibility tests are disabled for version 0.1.2.
=================== 3 failed, 29 passed, 5 skipped in 22.84s ===================

arsenlosenko · 2022-09-06T09:53:49Z

/publish connector=connectors/source-pinterest run-tests=false

🕑 Publishing the following connectors:
connectors/source-pinterest
https://github.com/airbytehq/airbyte/actions/runs/2999206161

Connector	Did it publish?	Were definitions generated?
connectors/source-pinterest	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

sherifnada · 2022-09-07T02:48:35Z

@arsenlosenko why were tests skipped here?

sherifnada · 2022-09-07T02:50:46Z

airbyte-oauth/src/main/java/io/airbyte/oauth/flows/PinterestOAuthFlow.java

+/**
+ * Following docs from https://developers.pinterest.com/docs/getting-started/authentication
+ */
+public class PinterestOAuthFlow extends BaseOAuth2Flow {


@arsenlosenko could you add unit tests for this function?

@sherifnada yes, already working on unit tests for this class in scope of this PR:
#16393

arsenlosenko · 2022-09-07T12:30:35Z

@arsenlosenko why were tests skipped here?

@sherifnada the tests were skipped due to low rate limits in Pinterest API (300 requests/day), due to that we couldn't run complete test suite here

* Source Pinterest: Initial setup of OAuth flow * Remove previously added class and method for auth * Update Java part of OAuth flow, update spec * Update spec and add additional methods to Java OAuth flow * Add backwards compatibility for old config structure * Add missing imports * Revert previous changes in source_specs.yaml * Cleanup in imports and source_specs * Add missing imports * Add missing imports * Remove 'subdomain' logic from Java OAuth flow * Update docs * Update docs accordingly to comments in PR * Refactor credentials variable * Fix typo * Update acceptance-test-config.yml * Specify integer type for AD_ACCOUNT_ID value in schemas * updated SAT tests, fixed Max Rate Limit error handling * updated unit_tests * updated schemas, added caching for Boards and AdAccounts stream to reduce API Calls for child streams, commented out Incremental and Full refresh SAT tests * auto-bump connector version [ci skip] Co-authored-by: Oleksandr Bazarnov <oleksandr.bazarnov@globallogic.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

YowanR · 2022-12-06T18:00:14Z

We got access to a higher rate limit for this account. Can we just run the test and make sure we are good now, please?
cc @arsenlosenko @lazebnyi

arsenlosenko requested review from bazarnov and roman-yermilov-gl September 2, 2022 13:33

arsenlosenko self-assigned this Sep 2, 2022

github-actions bot added the area/connectors Connector related issues label Sep 2, 2022

octavia-squidington-iii added the connectors/source/pinterest label Sep 2, 2022

arsenlosenko mentioned this pull request Sep 2, 2022

Source Pinterest: support OAuth #16165

Closed

37 tasks

arsenlosenko linked an issue Sep 2, 2022 that may be closed by this pull request

Source Pinterest: support oauth #7754

Closed

arsenlosenko temporarily deployed to more-secrets September 2, 2022 13:34 Inactive

arsenlosenko requested a review from lazebnyi September 2, 2022 14:28

github-actions bot added the area/documentation Improvements or additions to documentation label Sep 2, 2022

arsenlosenko temporarily deployed to more-secrets September 2, 2022 14:30 Inactive

bazarnov requested changes Sep 2, 2022

View reviewed changes

arsenlosenko temporarily deployed to more-secrets September 2, 2022 16:56 Inactive

Arsen Losenko added 13 commits September 2, 2022 19:57

Source Pinterest: Initial setup of OAuth flow

004f409

Remove previously added class and method for auth

661764e

Update Java part of OAuth flow, update spec

dbd6d4c

Update spec and add additional methods to Java OAuth flow

8f3774b

Add backwards compatibility for old config structure

f98e8c9

Add missing imports

3b695ad

Revert previous changes in source_specs.yaml

6ba6f4d

Cleanup in imports and source_specs

14df38b

Add missing imports

cd29dee

Add missing imports

7bb73a8

Remove 'subdomain' logic from Java OAuth flow

1ed60a3

Update docs

1a25c28

Update docs accordingly to comments in PR

739fb0f

arsenlosenko force-pushed the arsenlosenko/7754-source-pinterest-support-oauth-cp branch from 9743a03 to 739fb0f Compare September 2, 2022 16:59

arsenlosenko temporarily deployed to more-secrets September 2, 2022 17:01 Inactive

Specify integer type for AD_ACCOUNT_ID value in schemas

6405cc1

arsenlosenko temporarily deployed to more-secrets September 3, 2022 11:18 Inactive

updated SAT tests, fixed Max Rate Limit error handling

0c7595e

bazarnov temporarily deployed to more-secrets September 4, 2022 17:04 Inactive

Merge remote-tracking branch 'origin/master' into arsenlosenko/7754-s…

eff9fe1

…ource-pinterest-support-oauth-cp

bazarnov temporarily deployed to more-secrets September 4, 2022 17:08 Inactive

updated unit_tests

7786d8d

bazarnov temporarily deployed to more-secrets September 4, 2022 18:01 Inactive

bazarnov added 2 commits September 5, 2022 12:44

updated schemas, added caching for Boards and AdAccounts stream to re…

f9b320b

…duce API Calls for child streams, commented out Incremental and Full refresh SAT tests

Merge remote-tracking branch 'origin/master' into arsenlosenko/7754-s…

1019b9b

…ource-pinterest-support-oauth-cp

bazarnov temporarily deployed to more-secrets September 5, 2022 09:48 Inactive

lazebnyi approved these changes Sep 5, 2022

View reviewed changes

lazebnyi requested a review from bazarnov September 5, 2022 19:12

bazarnov approved these changes Sep 5, 2022

View reviewed changes

auto-bump connector version [ci skip]

deba310

arsenlosenko merged commit 46112cf into master Sep 6, 2022

arsenlosenko deleted the arsenlosenko/7754-source-pinterest-support-oauth-cp branch September 6, 2022 10:16

sherifnada reviewed Sep 7, 2022

View reviewed changes

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Source Pinterest: support OAuth #16271

Source Pinterest: support OAuth #16271

arsenlosenko commented Sep 2, 2022

CLAassistant commented Sep 2, 2022 •

edited

Loading

arsenlosenko commented Sep 2, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 3, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 4, 2022 •

edited by github-actions bot

Loading

bazarnov commented Sep 5, 2022 •

edited by github-actions bot

Loading

bazarnov commented Sep 5, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 6, 2022 •

edited by github-actions bot

Loading

sherifnada commented Sep 7, 2022

sherifnada Sep 7, 2022

arsenlosenko Sep 7, 2022 •

edited

Loading

arsenlosenko commented Sep 7, 2022

YowanR commented Dec 6, 2022 •

edited

Loading

Source Pinterest: support OAuth #16271

Source Pinterest: support OAuth #16271

Conversation

arsenlosenko commented Sep 2, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

CLAassistant commented Sep 2, 2022 • edited Loading

arsenlosenko commented Sep 2, 2022 • edited by github-actions bot Loading

Build Failed

arsenlosenko commented Sep 3, 2022 • edited by github-actions bot Loading

Build Failed

arsenlosenko commented Sep 4, 2022 • edited by github-actions bot Loading

Build Failed

bazarnov commented Sep 5, 2022 • edited by github-actions bot Loading

Build Failed

bazarnov commented Sep 5, 2022 • edited by github-actions bot Loading

Build Failed

arsenlosenko commented Sep 6, 2022 • edited by github-actions bot Loading

sherifnada commented Sep 7, 2022

sherifnada Sep 7, 2022

Choose a reason for hiding this comment

arsenlosenko Sep 7, 2022 • edited Loading

Choose a reason for hiding this comment

arsenlosenko commented Sep 7, 2022

YowanR commented Dec 6, 2022 • edited Loading

CLAassistant commented Sep 2, 2022 •

edited

Loading

arsenlosenko commented Sep 2, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 3, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 4, 2022 •

edited by github-actions bot

Loading

bazarnov commented Sep 5, 2022 •

edited by github-actions bot

Loading

bazarnov commented Sep 5, 2022 •

edited by github-actions bot

Loading

arsenlosenko commented Sep 6, 2022 •

edited by github-actions bot

Loading

arsenlosenko Sep 7, 2022 •

edited

Loading

YowanR commented Dec 6, 2022 •

edited

Loading