[low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file #18532

brianjlai · 2022-10-27T06:34:59Z

What

Right now, connectors built using the low-code framework require that each stream have a schema defined in the schemas/<stream_schema>.json. For low-code connector iterative development, schemas are not strictly necessary for read. To make the Connector Builder UI scope smaller, this change allows read to be performed w/o schemas

How

Defines a MockSchemaLoader which returns an empty schema back to the underlying CDK. This MockSchemaLoader is the new default for low code connectors that don't define a schema_loader. New connectors built in the Connector Builder UI will not need schemas defined to run read.
Renamed JsonSchema to JsonFileSchemaLoader for better clarity
We don't have a great versioning story, but we only have a few low code connectors, so I went ahead and migrated all of them to the new name. I can redeploy them after I publish the CDK.

Recommended reading order

mock_schema_loader.py
json_file_schema_loader.py
declarative_stream.py
The existing low code connector yaml that are switching to the new name

…SchemaLoader

sherifnada

would this cause all discover operations to return empty schemas unless the developer specifically specifies a Json schema loader for the connector? if so why not wrap the json loader in a "fallback" or "EmptyOnError" loader or something, which defaults to empty if the loader it wraps fails?

brianjlai · 2022-10-27T21:43:35Z

would this cause all discover operations to return empty schemas unless the developer specifically specifies a Json schema loader for the connector? if so why not wrap the json loader in a "fallback" or "EmptyOnError" loader or something, which defaults to empty if the loader it wraps fails?

Given our discussion during planning specifically that an empty schema would be a part of the initial development flow before exporting the YAML and adding schemas before running source acceptance tests, we should be okay on this point

sherifnada

non blocking coments

sherifnada · 2022-10-27T21:54:46Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py

    "ListStreamSlicer": ListStreamSlicer,
    "MinMaxDatetime": MinMaxDatetime,
+    "MockSchemaLoader": MockSchemaLoader,


suggestion: I'd replace Mock with Empty in the name

sherifnada · 2022-10-27T21:55:36Z

airbyte-cdk/python/airbyte_cdk/sources/declarative/parsers/class_types_registry.py

@@ -62,9 +63,10 @@
    "HttpRequester": HttpRequester,
    "InterpolatedBoolean": InterpolatedBoolean,
    "InterpolatedString": InterpolatedString,
-    "JsonSchema": JsonSchema,


is there a worry that this might break existing users?

I scanned through our existing low code implementations and i'm going to migrate them to the new name JsonFileSchemaLoader after the CDK is published first. However, on second thought, I think instead of doing the rename now, we'll just bundle this up into a series of cosmetic changes we have slated in November. Even though Hacktober fest is over, some contributors are still making adjustment and we don't want to add unnecessary friction if it can wait

brianjlai · 2022-11-03T03:00:39Z

/publish-cdk dry-run=true

🕑 https://github.com/airbytehq/airbyte/actions/runs/3382684219
✅ https://github.com/airbytehq/airbyte/actions/runs/3382684219

brianjlai · 2022-11-03T05:11:20Z

/publish-cdk dry-run=false

🕑 https://github.com/airbytehq/airbyte/actions/runs/3383169255
✅ https://github.com/airbytehq/airbyte/actions/runs/3383169255

brianjlai · 2022-11-03T15:19:20Z

/test connector=connectors/source-greenhouse

🕑 connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/3386946834
❌ connectors/source-greenhouse https://github.com/airbytehq/airbyte/actions/runs/3386946834
🐛 https://gradle.com/s/njbewgfnia5pw

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_streams.py::test_ignore_403 - requests.exceptions.JSON...
	 �[31m================= �[31m�[1m1 failed�[0m, �[32m10 passed�[0m, �[33m1520 warnings�[0m�[31m in 53.94s�[0m�[31m =================�[0m

brianjlai · 2022-11-03T15:20:21Z

/test connector=connectors/source-courier

🕑 connectors/source-courier https://github.com/airbytehq/airbyte/actions/runs/3386955355
✅ connectors/source-courier https://github.com/airbytehq/airbyte/actions/runs/3386955355
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       133      3    98%   87, 93, 230
	 source_acceptance_test/conftest.py                     196     97    51%   35, 41-43, 48, 54, 60, 66, 72-74, 80-95, 100, 105-107, 113-115, 121-122, 127-128, 133, 139, 148-157, 163-168, 232, 238, 244-250, 258-263, 271-284, 289-295, 302-313, 320-336
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              329    106    68%   39, 50-58, 63-70, 74-75, 79-80, 164, 202-219, 228-236, 240-245, 251, 284-289, 327-334, 377-379, 382, 447-455, 484-485, 491, 494, 530-540, 553-578
	 source_acceptance_test/tests/test_incremental.py       145     20    86%   21-23, 29-31, 36-43, 48-61, 224
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  77     10    87%   15-16, 24-30, 64, 67
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/config_migration.py        23     23     0%   5-37
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     105     13    88%   30-31, 38, 41, 65-68, 96, 120, 190-192
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1479    376    75%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============ 26 passed, 1 skipped, 27 warnings in 90.48s (0:01:30) =============

brianjlai · 2022-11-03T15:47:42Z

note: greenhouse has an existing test failure

… streams w/o a schema file (#18532) * use mockschemaloader as the default and rename JsonSchema to JsonFileSchemaLoader * rename mock to EmptySchemaLoader * retain existing JsonSchema while hacktoberfest is still in progress * bump version and changelog

… streams w/o a schema file (airbytehq#18532) * use mockschemaloader as the default and rename JsonSchema to JsonFileSchemaLoader * rename mock to EmptySchemaLoader * retain existing JsonSchema while hacktoberfest is still in progress * bump version and changelog

use mockschemaloader as the default and rename JsonSchema to JsonFile…

8ead873

…SchemaLoader

github-actions bot added area/connectors Connector related issues CDK Connector Development Kit labels Oct 27, 2022

octavia-squidington-iv added connectors/source/sentry connectors/source/greenhouse connectors/source/sendgrid connectors/source/courier connectors/source/gocardless labels Oct 27, 2022

brianjlai requested a review from girarda October 27, 2022 06:36

brianjlai marked this pull request as ready for review October 27, 2022 06:37

brianjlai requested a review from a team as a code owner October 27, 2022 06:37

sherifnada reviewed Oct 27, 2022

View reviewed changes

sherifnada approved these changes Oct 27, 2022

View reviewed changes

girarda approved these changes Nov 1, 2022

View reviewed changes

brianjlai added 4 commits November 2, 2022 17:39

Merge branch 'master' into brian/low_code_read_without_schema_file

0acada0

rename mock to EmptySchemaLoader

71b2b58

retain existing JsonSchema while hacktoberfest is still in progress

f23e923

bump version and changelog

7b9a889

brianjlai merged commit cf2f23d into master Nov 3, 2022

brianjlai deleted the brian/low_code_read_without_schema_file branch November 3, 2022 16:05

brianjlai mentioned this pull request Nov 4, 2022

[low-code] replace emptySchemaLoader with DefaultSchemaLoader #18947

Merged

github-actions bot mentioned this pull request Nov 15, 2022

Bump helm chart version reference to 0.40.43 #19453

Closed

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file #18532

[low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file #18532

brianjlai commented Oct 27, 2022 •

edited

Loading

sherifnada left a comment

brianjlai commented Oct 27, 2022

sherifnada left a comment

sherifnada Oct 27, 2022

sherifnada Oct 27, 2022

brianjlai Nov 2, 2022

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022

[low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file #18532

[low-code cdk] Allow for read command to be run on low code connector streams w/o a schema file #18532

Conversation

brianjlai commented Oct 27, 2022 • edited Loading

What

How

Recommended reading order

sherifnada left a comment

Choose a reason for hiding this comment

brianjlai commented Oct 27, 2022

sherifnada left a comment

Choose a reason for hiding this comment

sherifnada Oct 27, 2022

Choose a reason for hiding this comment

sherifnada Oct 27, 2022

Choose a reason for hiding this comment

brianjlai Nov 2, 2022

Choose a reason for hiding this comment

brianjlai commented Nov 3, 2022 • edited by github-actions bot Loading

brianjlai commented Nov 3, 2022 • edited by github-actions bot Loading

brianjlai commented Nov 3, 2022 • edited by github-actions bot Loading

Build Failed

brianjlai commented Nov 3, 2022 • edited by github-actions bot Loading

Build Passed

brianjlai commented Nov 3, 2022

brianjlai commented Oct 27, 2022 •

edited

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading

brianjlai commented Nov 3, 2022 •

edited by github-actions bot

Loading