Make schema field in source-snowflake mean a subset of the specified o… #20465

rodireich · 2022-12-14T07:32:27Z

What

the schema field in source-snowflake is not very useful today:
Regardless of what the user inputs in schema, all tables from all schemas are discovered and included in catalog.
This change changes its behavior to be more in line with other source connectors.

How

If schema is specified, the catalog discovery will be limited to only tables included in this schema.
If field left open, all tables from all schema will be included in catalog - similarly to today's discovery.

🚨 User Impact 🚨

This is going to have in impact, as source-snowflake connectors today have a mandatory schema field, but replication might include tables from outside the specified schema.
These customers needs to be instructed to leave the schema field empty upon upgrade (field is mandatory today).
Existing connection that only sync tables from the specified schema should not have a problem.

>>>>>>>> We need to be carful in publishing this in order not to break existing connections

Do not merge until we have a go ahead from TCS!!

cc: @erica-airbyte

…f schema when during discover(). update UI

github-actions · 2022-12-14T07:34:02Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

❌ Sources (30)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.34`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.34`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.14`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.14`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.18`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.18`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.16`	✅	✅
`source-db2-strict-encrypt`	`0.1.16`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.0`	✅	✅
`source-e2e-test`	`2.1.3`	✅	✅
`source-e2e-test-cloud`	`2.1.1`	🔵 (ignored)	🔵 (ignored)
`source-elasticsearch`	`0.1.1`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-kafka`	`0.2.3`	✅	✅
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`0.4.26`	✅	✅
`source-mssql-strict-encrypt`	`0.4.26`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`1.0.18`	✅	✅
`source-mysql-strict-encrypt`	`1.0.18`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.21`	✅	✅
`source-oracle-strict-encrypt`	`0.3.21`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`1.0.35`	✅	✅
`source-postgres-strict-encrypt`	`1.0.35`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.15`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-sftp`	`0.1.2`	✅	✅
`source-snowflake`	`0.1.28`	✅	❌ (diff seed version)
`source-tidb`	`0.2.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

rodireich · 2022-12-14T23:11:50Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3699515235
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3699515235
🐛 https://gradle.com/s/gstpokxsslwiw

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-12-14T23:22:10Z

...rc/main/java/io/airbyte/integrations/standardtest/source/AbstractSourceDatabaseTypeTest.java

-        .collect(Collectors.toMap(AirbyteStream::getName, s -> s));
+        .collect(Collectors.toMap(
+            s ->
+                "%s.%s".formatted(s.getNamespace(), s.getName()),


This change makes the test more robust in case previous runs left junk schemas or in case multiple instances of acceptance test are running at the same time.

rodireich · 2022-12-14T23:54:56Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3699740148
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3699740148
🐛 https://gradle.com/s/i5tpwjwha5k6q

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
============== 28 passed, 1 skipped, 1 error in 62.30s (0:01:02) ===============

rodireich · 2022-12-15T03:21:19Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3700775419

rodireich · 2022-12-15T03:42:23Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3700872496
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3700872496
🐛 https://gradle.com/s/hvkpfoqcklumy

Build Failed

Test summary info:

=========================== short test summary info ============================
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
============== 28 passed, 1 skipped, 1 error in 63.08s (0:01:03) ===============

rodireich · 2022-12-15T05:29:03Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3701376021
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3701376021
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:26: `future_state` not specified, skipping.
=================== 29 passed, 1 skipped in 70.39s (0:01:10) ===================

rodireich · 2022-12-15T06:09:53Z

airbyte-config/init/src/test/resources/connector_catalog.json

@@ -12691,7 +12691,7 @@
          "$schema": "http://json-schema.org/draft-07/schema#",


@grishick , @evantahler Do we need to make changes to connector_catalog as well?

…t-used

…-schema-not-used' into 20018-snowflake-source-connector-schema-not-used

rodireich · 2022-12-16T19:58:27Z

Going over metabase I'm seeing the following active cloud source-snowflake connections:

40ea7f191a87: Only syncing tables of specified schema. Won't break ✔
5588f84cedd0: Won't break ✔
8cb1f1336d6d: Won't break ✔
1109283f47c7: Won't break ✔
d845f68fa43c: Won't break ✔
e34c23e109cf: Won't break ✔

rodireich · 2023-01-06T06:44:45Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3853261129
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3853261129
🐛 https://gradle.com/s/a54klqihmqlw4

Build Failed

Test summary info:

Could not find result summary

rodireich · 2023-01-06T07:18:51Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3853437386
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3853437386
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   56-63, 68-81, 244
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:30: `future_state` not specified, skipping.
=================== 28 passed, 2 skipped in 72.17s (0:01:12) ===================

rodireich · 2023-01-06T07:37:32Z

/publish connector=connectors/source-snowflake

🕑 Publishing the following connectors:
connectors/source-snowflake
https://github.com/airbytehq/airbyte/actions/runs/3853527039

Connector	Did it publish?	Were definitions generated?
connectors/source-snowflake	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…t-used

…o… (#20465) * Make schema field in source-postgres mean a subset of the specified of schema when during discover(). update UI * Add missing file * Fix failing acceptance test * sanity * update doc * typo * version bump and release note * Fix failing test * fix format

sheshan-doye-konvergeai · 2023-02-27T10:21:01Z

trying snowflake 1.28 source version,
it seems now nothing is discovered if schema is provided
response:
{"catalog":{"streams":[]},"jobInfo":{"id":"c2af2abf-05c8-4525-aea2-ec10002a6832","configType":"discover_schema","configId":"Optional[e2d65910-8c8b-40a1-ae7d-ee2416b2bfa2]","createdAt":1677493097784,"endedAt":1677493103991,"succeeded":true

am i missing anything?

Make schema field in source-postgres mean a subset of the specified o…

cb9138b

…f schema when during discover(). update UI

rodireich linked an issue Dec 14, 2022 that may be closed by this pull request

Source Snowflake - connector Schema not used #20018

Closed

octavia-squidington-iv added area/connectors Connector related issues connectors/source/snowflake labels Dec 14, 2022

Add missing file

cad51d3

rodireich commented Dec 14, 2022

View reviewed changes

Fix failing acceptance test

116be53

rodireich temporarily deployed to more-secrets December 15, 2022 03:43 — with GitHub Actions Inactive

rodireich commented Dec 15, 2022

View reviewed changes

rodireich requested review from subodh1810, sashaNeshcheret, VitaliiMaltsev, akashkulk and grishick December 15, 2022 06:21

Merge branch 'master' into 20018-snowflake-source-connector-schema-no…

4bedc60

…t-used

rodireich temporarily deployed to more-secrets December 15, 2022 06:25 — with GitHub Actions Inactive

rodireich added 2 commits December 14, 2022 22:27

sanity

55df832

Merge remote-tracking branch 'origin/20018-snowflake-source-connector…

85dbea6

…-schema-not-used' into 20018-snowflake-source-connector-schema-not-used

rodireich temporarily deployed to more-secrets December 15, 2022 06:30 — with GitHub Actions Inactive

update doc

49273e8

rodireich temporarily deployed to more-secrets January 6, 2023 05:56 — with GitHub Actions Inactive

rodireich temporarily deployed to more-secrets January 6, 2023 05:57 — with GitHub Actions Inactive

version bump and release note

3ce98b7

rodireich temporarily deployed to more-secrets January 6, 2023 06:07 — with GitHub Actions Inactive

Fix failing test

898591b

rodireich temporarily deployed to more-secrets January 6, 2023 07:20 — with GitHub Actions Inactive

rodireich temporarily deployed to more-secrets January 6, 2023 07:21 — with GitHub Actions Inactive

fix format

0819c8b

rodireich temporarily deployed to more-secrets January 6, 2023 07:44 — with GitHub Actions Inactive

rodireich temporarily deployed to more-secrets January 6, 2023 07:45 — with GitHub Actions Inactive

rodireich temporarily deployed to more-secrets January 6, 2023 08:05 — with GitHub Actions Inactive

Merge branch 'master' into 20018-snowflake-source-connector-schema-no…

855bc4b

…t-used

rodireich temporarily deployed to more-secrets January 6, 2023 09:02 — with GitHub Actions Inactive

Merge branch 'master' into 20018-snowflake-source-connector-schema-no…

a242f66

…t-used

rodireich temporarily deployed to more-secrets January 11, 2023 17:14 — with GitHub Actions Inactive

subodh1810 approved these changes Jan 12, 2023

View reviewed changes

rodireich merged commit 94c84f7 into master Jan 12, 2023

rodireich deleted the 20018-snowflake-source-connector-schema-not-used branch January 12, 2023 17:30

rodireich mentioned this pull request Jan 12, 2023

manually bump source definition version #21349

Merged

octavia-squidington-iii mentioned this pull request Jan 12, 2023

Bump Airbyte version from 0.40.27 to 0.40.28 #21359

Merged

sheshan-doye-konvergeai mentioned this pull request Feb 27, 2023

Source Snowflake schema not in effect #23469

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make schema field in source-snowflake mean a subset of the specified o… #20465

Make schema field in source-snowflake mean a subset of the specified o… #20465

rodireich commented Dec 14, 2022 •

edited

Loading

github-actions bot commented Dec 14, 2022 •

edited

Loading

❌ Sources (30)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

rodireich commented Dec 14, 2022 •

edited by github-actions bot

Loading

rodireich Dec 14, 2022 •

edited

Loading

rodireich commented Dec 14, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich Dec 15, 2022

rodireich commented Dec 16, 2022 •

edited

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

sheshan-doye-konvergeai commented Feb 27, 2023 •

edited

Loading

		@@ -12691,7 +12691,7 @@
		"$schema": "http://json-schema.org/draft-07/schema#",

Make schema field in source-snowflake mean a subset of the specified o… #20465

Make schema field in source-snowflake mean a subset of the specified o… #20465

Conversation

rodireich commented Dec 14, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Do not merge until we have a go ahead from TCS!!

github-actions bot commented Dec 14, 2022 • edited Loading

Affected Connector Report

❌ Sources (30)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

rodireich commented Dec 14, 2022 • edited by github-actions bot Loading

Build Failed

rodireich Dec 14, 2022 • edited Loading

Choose a reason for hiding this comment

rodireich commented Dec 14, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 15, 2022 • edited by github-actions bot Loading

rodireich commented Dec 15, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 15, 2022 • edited by github-actions bot Loading

Build Passed

rodireich Dec 15, 2022

Choose a reason for hiding this comment

rodireich commented Dec 16, 2022 • edited Loading

rodireich commented Jan 6, 2023 • edited by github-actions bot Loading

Build Failed

rodireich commented Jan 6, 2023 • edited by github-actions bot Loading

Build Passed

rodireich commented Jan 6, 2023 • edited by github-actions bot Loading

sheshan-doye-konvergeai commented Feb 27, 2023 • edited Loading

rodireich commented Dec 14, 2022 •

edited

Loading

github-actions bot commented Dec 14, 2022 •

edited

Loading

rodireich commented Dec 14, 2022 •

edited by github-actions bot

Loading

rodireich Dec 14, 2022 •

edited

Loading

rodireich commented Dec 14, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 15, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 16, 2022 •

edited

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

rodireich commented Jan 6, 2023 •

edited by github-actions bot

Loading

sheshan-doye-konvergeai commented Feb 27, 2023 •

edited

Loading