🎉 New Source: Pexels API [low-code cdk] #18854

btkcodedev · 2022-11-02T16:03:43Z

What

Developing new connector for the source: Pexels API
New data source will be definitely useful for image data url's

Resolves:
🎉 New Source: Pexels API

How

Developed using low-code cdk

🚨 User Impact 🚨

No breaking changes, Just addition of new source

New Connector: Source Pexels API

Community member or Airbyter

Tests

Integration & Acceptance

Full Test Results

marcosmarxm

Hello @btkcodedev, Marcos from Airbyte here 👋 . We received more than 25 new contributions along the weekend. One is yours 🎉 thank so much for! Our team is limited and maybe the review process can take longer than expected. As described in the Airbyte's Hacktoberfest your contribution was submitted before November 2nd and it is eligible to win the prize. The review process will validate other requirements. I ask to you patience until someone from the team review it.

Because I reviewed some contributions for Hacktoberfest so far I saw some common patterns you can check in advance:

Make sure you have added connector documentation to /docs/integrations/
Remove the file catalog from /integration_tests
Edit the sample_config.json inside /integration_tests
For the configured_catalog you can use only json_schema: {}
Add title to all properties in the spec.yaml
Make sure the documentationUrl in the spec.yaml redirect to Airbyte's future connector page, eg: connector Airtable the documentationUrl: https://docs.airbyte.com/integrations/sources/airtable
Review now new line at EOF (end-of-file) for all files.

If possible send to me a DM in Slack with the tests credentials, this process will make easier to us run integration tests and publish your connector. If you only has production keys, make sure to create a bootstrap.md explaining how to get the keys.

marcosmarxm · 2022-11-04T08:56:35Z

Hello! I'm going to be out of the office this Friday and won't be able to review your contribution again today, I return next Monday. So far, most contributions look solid and are almost done to be approved. As said in Chris' comment all contributions made before 2-November are eligible to receive the prize and have 2 weeks to merge the contributions. But I ensure next week we're going to have your contribution merged. If you have questions about the implementation you can send them in #hacktoberfest-2022 in Airbyte's Slack.

Sorry the inconvenience and see you again next week, thank you so much for your contribution!

btkcodedev · 2022-11-05T19:43:33Z

All comments resolved.
@marcosmarxm Review Required from community.

marcosmarxm

Some comments, there are some points need imprvements.

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/spec.yaml

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/pexels_api.yaml

btkcodedev · 2022-11-09T16:31:13Z

@marcosmarxm All comments resolved.
Test results

marcosmarxm

Some comments.

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/pexels_api.yaml

btkcodedev · 2022-11-10T16:00:35Z

My config:
{
"api_key": "563492ad6f9170000100000126081f5bb22a4748b3575c3af4540e2f",
"query":"people",
"orientation": "landscape",
"size": "large",
"color": "orange",
"locale": "en-US",
"photo_id": "3756777",
"video_id": "2499611",
"collection_id": "mgnr9pi"
}

Recent Test Result:

btkcodedev · 2022-11-10T16:08:24Z

@marcosmarxm All comments resolved

marcosmarxm · 2022-11-14T13:48:06Z

@btkcodedev I'll review your PR today!

marcosmarxm

One comment, I'm having problem running tests with current configuration.

marcosmarxm · 2022-11-14T23:13:39Z

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/spec.yaml

+    - photo_id
+    - video_id
+    - collection_id


@btkcodedev I think this still not right. I don't want to retrieve only one photo and a video and a collection. I want to probably pass a collection id and retrieve all data inside of it.
I think most valuable for this connector it is to only implement the endpoint for search or retrieve data from collections id.

marcosmarxm · 2022-11-18T16:28:03Z

Because this contribution didn't receive recent updates it won't be consider valid to hacktoberfest event. You can continue the work to get this merge in your own pace. Thank so much for your contribution!

CLAassistant · 2022-11-18T18:31:31Z

All committers have signed the CLA.

btkcodedev · 2022-11-18T18:33:08Z

um..I've just made the final change just right now. It's ready for shipping
@marcosmarxm It's your call boss, You could give an exception or just be it out.
Either way, i will be happy with the decision :)

marcosmarxm · 2022-11-18T18:35:11Z

I'll test your changes later.

btkcodedev · 2022-11-18T18:39:56Z

Thanks! pal. Really appreciating your maintainership
My config:
{
"api_key": "563492ad6f9170000100000126081f5bb22a4748b3575c3af4540e2f",
"query":"people",
"orientation": "landscape",
"size": "large",
"color": "orange",
"locale": "en-US"
}

marcosmarxm · 2022-11-18T20:35:34Z

/test connector=connectors/source-pexels-api

🕑 connectors/source-pexels-api https://github.com/airbytehq/airbyte/actions/runs/3500061852
❌ connectors/source-pexels-api https://github.com/airbytehq/airbyte/actions/runs/3500061852
🐛 https://gradle.com/s/ed6o4k3kjfgio

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs1] - Failed: Timeout >3...
FAILED test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contai...
FAILED test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
=================== 3 failed, 21 passed, 3 skipped in 54.86s ===================

marcosmarxm · 2022-11-18T22:23:39Z

/test connector=connectors/source-pexels-api

🕑 connectors/source-pexels-api https://github.com/airbytehq/airbyte/actions/runs/3500652290
✅ connectors/source-pexels-api https://github.com/airbytehq/airbyte/actions/runs/3500652290
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:88: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:364: The previous connector image could not be retrieved.
======================== 26 passed, 3 skipped in 29.59s ========================

marcosmarxm · 2022-11-18T22:40:17Z

/publish connector=connectors/source-pexels-api

🕑 Publishing the following connectors:
connectors/source-pexels-api
https://github.com/airbytehq/airbyte/actions/runs/3500735001

Connector	Did it publish?	Were definitions generated?
connectors/source-pexels-api	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-11-19T00:25:59Z

well well, @RealChrisSean base branch is locked because of code freeze. I can't merge this PR but it was published before the deadline!

Thanks for the contribution @btkcodedev

marcosmarxm · 2022-11-22T20:24:44Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

btkcodedev · 2022-11-26T17:48:38Z

@marcosmarxm Could I get that hacktober label back and merge this PR :)

New Source: Pexels API

032dd44

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Nov 2, 2022

octavia-squidington-iv added bounty community connectors/source/pexels-api labels Nov 2, 2022

btkcodedev mentioned this pull request Nov 2, 2022

🎉New Source: Pexels API airbytehq/connector-contest#250

Closed

marcosmarxm added the hacktober label Nov 2, 2022

marcosmarxm requested changes Nov 2, 2022

View reviewed changes

Merge branch 'master' into pexels-api

5d20ea4

Resolved all comments

365155b

btkcodedev requested a review from marcosmarxm November 5, 2022 19:43

marcosmarxm requested changes Nov 7, 2022

View reviewed changes

feat: Added paginator, refactor: moved params to request options

2f6c10e

btkcodedev requested a review from marcosmarxm November 9, 2022 16:31

marcosmarxm requested changes Nov 10, 2022

View reviewed changes

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/pexels_api.yaml Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-pexels-api/source_pexels_api/pexels_api.yaml Outdated Show resolved Hide resolved

refactor: configured params, moved request option to referenced

72f7163

btkcodedev requested a review from marcosmarxm November 10, 2022 16:01

marcosmarxm requested changes Nov 14, 2022

View reviewed changes

marcosmarxm removed the hacktober label Nov 18, 2022

Corrected comments

0ad2eb8

btkcodedev requested review from marcosmarxm and removed request for marcosmarxm November 18, 2022 18:33

btkcodedev removed the request for review from marcosmarxm November 18, 2022 18:35

correct test and add pexels api to source def

b5898f0

Merge branch 'master' into pexels-api

f412ded

octavia-squidington-iii and others added 2 commits November 18, 2022 22:59

auto-bump connector version

54c1114

Merge branch 'master' into pexels-api

45e23d8

marcosmarxm approved these changes Nov 19, 2022

View reviewed changes

Merge branch 'master' into pexels-api

be41c47

btkcodedev requested a review from marcosmarxm November 29, 2022 17:36

Merge branch 'master' into pexels-api

f854cdb

marcosmarxm approved these changes Nov 29, 2022

View reviewed changes

marcosmarxm merged commit db836b3 into airbytehq:master Nov 29, 2022

octavia-squidington-iii mentioned this pull request Nov 30, 2022

Bump Airbyte version from 0.40.22 to 0.40.23 #19941

Merged

sajarin added internal and removed bounty labels Dec 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: Pexels API [low-code cdk] #18854

🎉 New Source: Pexels API [low-code cdk] #18854

btkcodedev commented Nov 2, 2022 •

edited

Loading

marcosmarxm left a comment

marcosmarxm commented Nov 4, 2022

btkcodedev commented Nov 5, 2022

marcosmarxm left a comment

btkcodedev commented Nov 9, 2022

marcosmarxm left a comment

btkcodedev commented Nov 10, 2022

btkcodedev commented Nov 10, 2022

marcosmarxm commented Nov 14, 2022

marcosmarxm left a comment

marcosmarxm Nov 14, 2022

marcosmarxm commented Nov 18, 2022

CLAassistant commented Nov 18, 2022 •

edited

Loading

btkcodedev commented Nov 18, 2022

marcosmarxm commented Nov 18, 2022

btkcodedev commented Nov 18, 2022

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 19, 2022

marcosmarxm commented Nov 22, 2022

btkcodedev commented Nov 26, 2022

🎉 New Source: Pexels API [low-code cdk] #18854

🎉 New Source: Pexels API [low-code cdk] #18854

Conversation

btkcodedev commented Nov 2, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Community member or Airbyter

Tests

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm commented Nov 4, 2022

btkcodedev commented Nov 5, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

btkcodedev commented Nov 9, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

btkcodedev commented Nov 10, 2022

btkcodedev commented Nov 10, 2022

marcosmarxm commented Nov 14, 2022

marcosmarxm left a comment

Choose a reason for hiding this comment

marcosmarxm Nov 14, 2022

Choose a reason for hiding this comment

marcosmarxm commented Nov 18, 2022

CLAassistant commented Nov 18, 2022 • edited Loading

btkcodedev commented Nov 18, 2022

marcosmarxm commented Nov 18, 2022

btkcodedev commented Nov 18, 2022

marcosmarxm commented Nov 18, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Nov 18, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Nov 18, 2022 • edited by github-actions bot Loading

marcosmarxm commented Nov 19, 2022

marcosmarxm commented Nov 22, 2022

btkcodedev commented Nov 26, 2022

btkcodedev commented Nov 2, 2022 •

edited

Loading

CLAassistant commented Nov 18, 2022 •

edited

Loading

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 18, 2022 •

edited by github-actions bot

Loading