🎉 New Source: SpaceX API [low-code cdk] #18311

btkcodedev · 2022-10-21T17:35:53Z

What

Developing new connector for the source: SpaceX API
New data source with couple of endpoints will be definitely useful

Resolves:
🎉 New Source: SpaceX API

How

Developed using low-code cdk

🚨 User Impact 🚨

No breaking changes, Just addition of new source

New Connector: Source SpaceX API

Community member or Airbyter

Tests

Integration & Acceptance

Full Test Results

CLAassistant · 2022-10-21T17:35:58Z

All committers have signed the CLA.

btkcodedev · 2022-10-21T17:42:42Z

Hello maintainers. I am actually a newbie,
In need of help to proceed.

Unit test output are failing with 3 errors. Need feedback and help:
I've tested with one endpoint 'launches/latest'. And connection is succeeded and data is read.
But the test are failing with config.json and catalog_config.json
The API didn't use any auth, but i pass some dummy api key in header for future enhancements
I really can't understand about what is failing in acceptance test. I've corrected the schema and crosschecked everything.

NEED HELP

Edit: Problem solved

marcosmarxm · 2022-10-21T17:49:48Z

Thanks for the contribution @btkcodedev can you check your commit? There are some duplicate files.

dainiussa · 2022-10-21T22:02:50Z

airbyte-integrations/connectors/source-spacex-api/source_spacex_api/schemas/launches.json

+    },
+    "capsules": {
+      "type": "array",
+      "items": [


"items" must be an object instead of an array :)

@dainiussa Thanks for the help! Acceptance tests passed, Cheers!

btkcodedev · 2022-10-22T11:00:35Z

🐛Update: fix: schema is updated and matched with incoming record spec
Thus acceptance tests are resulted in passed state

Thanks

btkcodedev · 2022-10-22T11:03:56Z

TODO:

btkcodedev · 2022-10-22T13:15:11Z

@marcosmarxm Done, Duplicates removed

btkcodedev · 2022-10-22T14:22:47Z

Hello maintainers,
Need the "build URL" for this connector.
Can somebody help with that.

All other things are done.
@marcosmarxm

.vscode/settings.json

airbyte-integrations/connectors/source-spacex-api/source_spacex_api/spacex_api.yaml

btkcodedev · 2022-10-25T17:14:51Z

Ingested all endpoints
Launches config can now take options, id or keep it as null
Acceptance test gives pass result after all the changes to the connector

btkcodedev · 2022-10-25T17:15:55Z

@marcosmarxm All requested changes done. If the changes are fine, please proceed for the build.

marcosmarxm · 2022-10-25T23:45:44Z

Please check some left comments

btkcodedev · 2022-10-26T01:34:25Z

airbyte-integrations/connectors/source-spacex-api/source_spacex_api/spacex_api.yaml

+    $options:
+      name: "launches"
+      primary_key: "id"
+      path: "/launches/{{config['id'] or config['options'] or latest}}"


@marcosmarxm Config['options] are now provided so that latest, past, upcoming options could be configured if required.

btkcodedev · 2022-10-27T01:25:50Z

@marcosmarxm Acceptance test is passing after all changes.
Requesting review from the community.

Thanks :)

marcosmarxm · 2022-11-03T14:39:17Z

/test connector=connectors/source-spacex-api

🕑 connectors/source-spacex-api https://github.com/airbytehq/airbyte/actions/runs/3386628886
❌ connectors/source-spacex-api https://github.com/airbytehq/airbyte/actions/runs/3386628886
🐛 https://gradle.com/s/lpbgpjv6vx72q

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
============ 1 failed, 23 passed, 3 skipped, 27 warnings in 36.30s =============

marcosmarxm · 2022-11-03T17:55:41Z

@btkcodedev there is a test failing. I'll take a look tomorrow and publish the connector.

marcosmarxm · 2022-11-03T20:12:56Z

/test connector=connectors/source-spacex-api

🕑 connectors/source-spacex-api https://github.com/airbytehq/airbyte/actions/runs/3388975133
✅ connectors/source-spacex-api https://github.com/airbytehq/airbyte/actions/runs/3388975133
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       133      3    98%   87, 93, 230
	 source_acceptance_test/conftest.py                     196     97    51%   35, 41-43, 48, 54, 60, 66, 72-74, 80-95, 100, 105-107, 113-115, 121-122, 127-128, 133, 139, 148-157, 163-168, 232, 238, 244-250, 258-263, 271-284, 289-295, 302-313, 320-336
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              329    106    68%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 377-379, 382, 447-455, 484-485, 491, 494, 530-540, 553-578
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/config_migration.py        23     23     0%   5-37
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1479    376    75%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:51: The previous connector image could not be retrieved.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:229: The previous connector image could not be retrieved.
================= 24 passed, 3 skipped, 27 warnings in 36.34s ==================

marcosmarxm · 2022-11-03T23:25:07Z

/publish connector=connectors/source-spacex-api

🕑 Publishing the following connectors:
connectors/source-spacex-api
https://github.com/airbytehq/airbyte/actions/runs/3390054454

Connector	Did it publish?	Were definitions generated?
connectors/source-spacex-api	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

btkcodedev · 2022-11-04T03:12:16Z

Thanks @marcosmarxm, You're awesome.

RealChrisSean · 2022-11-18T07:52:29Z

Hello @btkcodedev can you please DM me your email for Paypal via slack :) Ty!

github-actions bot added the area/connectors Connector related issues label Oct 21, 2022

dainiussa reviewed Oct 21, 2022

View reviewed changes

btkcodedev changed the title ~~Init: New Source: SpaceX API~~ 🎉 New Source: SpaceX API Oct 22, 2022

github-actions bot added the area/documentation Improvements or additions to documentation label Oct 22, 2022

btkcodedev mentioned this pull request Oct 22, 2022

New Source: SpaceX API airbytehq/connector-contest#157

Closed

btkcodedev requested a review from dainiussa October 22, 2022 14:30

RealChrisSean added the hacktober label Oct 22, 2022

marcosmarxm added the community label Oct 24, 2022

marcosmarxm requested changes Oct 24, 2022

View reviewed changes

.vscode/settings.json Outdated Show resolved Hide resolved

airbyte-integrations/connectors/source-spacex-api/source_spacex_api/spacex_api.yaml Outdated Show resolved Hide resolved

bleonard added bounty connectors/source/spacex-api labels Oct 25, 2022

btkcodedev requested review from marcosmarxm and removed request for dainiussa October 25, 2022 17:16

btkcodedev commented Oct 26, 2022

View reviewed changes

btkcodedev requested review from dainiussa and marcosmarxm and removed request for marcosmarxm and dainiussa October 27, 2022 01:42

btkcodedev changed the title ~~🎉 New Source: SpaceX API~~ 🎉 New Source: SpaceX API [low-code cdk] Oct 27, 2022

btkcodedev and others added 14 commits November 3, 2022 11:21

chore: Created bootstrap.md

f8394df

chore: Update doc url, removed unwanted files

cf6d86a

fix: Fixed schema after crosscheck with several schema validators

203b7cf

fix: resolved .vscode setting conflict, resolved merge conflict

ba516a2

feat: Added support for all endpoints

a3b686a

chore: updated documentation

3873dde

feat: changes to stream path to accept options and id

aa3be6b

fix: Resolve coomments

23b5c22

chore: resolved comments

49ee57f

chore: Delete unwanted files

0c6df01

chore: Update Readme.md after resolving merge conflict

197ff70

update connector

bdf5713

add spacex api to source def

2f40be2

run format

f5a5909

marcosmarxm force-pushed the spacexapi branch from 46b7751 to f5a5909 Compare November 3, 2022 14:22

add schemaloader

136eb28

marcosmarxm and others added 2 commits November 3, 2022 19:25

Merge branch 'master' into spacexapi

dd9119c

auto-bump connector version

d53eabf

marcosmarxm merged commit 838aebe into airbytehq:master Nov 4, 2022

RealChrisSean added the low-code label Nov 15, 2022

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

sophia-wiley mentioned this pull request Mar 13, 2023

Docs: Update broken links #23940

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 New Source: SpaceX API [low-code cdk] #18311

🎉 New Source: SpaceX API [low-code cdk] #18311

btkcodedev commented Oct 21, 2022 •

edited

Loading

CLAassistant commented Oct 21, 2022 •

edited

Loading

btkcodedev commented Oct 21, 2022 •

edited

Loading

marcosmarxm commented Oct 21, 2022

dainiussa Oct 21, 2022

btkcodedev Oct 22, 2022 •

edited

Loading

btkcodedev commented Oct 22, 2022 •

edited

Loading

btkcodedev commented Oct 22, 2022 •

edited

Loading

btkcodedev commented Oct 22, 2022

btkcodedev commented Oct 22, 2022

btkcodedev commented Oct 25, 2022

btkcodedev commented Oct 25, 2022

marcosmarxm commented Oct 25, 2022

btkcodedev Oct 26, 2022

btkcodedev commented Oct 27, 2022

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 3, 2022

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading

btkcodedev commented Nov 4, 2022

RealChrisSean commented Nov 18, 2022

🎉 New Source: SpaceX API [low-code cdk] #18311

🎉 New Source: SpaceX API [low-code cdk] #18311

Conversation

btkcodedev commented Oct 21, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Community member or Airbyter

Tests

CLAassistant commented Oct 21, 2022 • edited Loading

btkcodedev commented Oct 21, 2022 • edited Loading

marcosmarxm commented Oct 21, 2022

dainiussa Oct 21, 2022

Choose a reason for hiding this comment

btkcodedev Oct 22, 2022 • edited Loading

Choose a reason for hiding this comment

btkcodedev commented Oct 22, 2022 • edited Loading

btkcodedev commented Oct 22, 2022 • edited Loading

btkcodedev commented Oct 22, 2022

btkcodedev commented Oct 22, 2022

btkcodedev commented Oct 25, 2022

btkcodedev commented Oct 25, 2022

marcosmarxm commented Oct 25, 2022

btkcodedev Oct 26, 2022

Choose a reason for hiding this comment

btkcodedev commented Oct 27, 2022

marcosmarxm commented Nov 3, 2022 • edited by github-actions bot Loading

Build Failed

marcosmarxm commented Nov 3, 2022

marcosmarxm commented Nov 3, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Nov 3, 2022 • edited by github-actions bot Loading

btkcodedev commented Nov 4, 2022

RealChrisSean commented Nov 18, 2022

btkcodedev commented Oct 21, 2022 •

edited

Loading

CLAassistant commented Oct 21, 2022 •

edited

Loading

btkcodedev commented Oct 21, 2022 •

edited

Loading

btkcodedev Oct 22, 2022 •

edited

Loading

btkcodedev commented Oct 22, 2022 •

edited

Loading

btkcodedev commented Oct 22, 2022 •

edited

Loading

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Nov 3, 2022 •

edited by github-actions bot

Loading