Code cleanup in `SourceOperations` #20874

akashkulk · 2022-12-23T21:47:48Z

Refactor includes:
JdbcCompatibleSourceOperations

rename setJsonField -> putJsonField
rename getFieldType -> getDatabaseFieldType
rename getJsonType -> getAirbyteType

AbstractJdbcCompatibleSourceOperations/JdbcCompatibleSourceOperations

move enquoteIdentifierList, enquoteIdentifier, getFullyQualifiedTableName, getFullQualifiedTableNameWithQuoting to
RelationalDbQueryUtil class

AbstractDbSource

rename getType -> getAirbyteType

Additional documentation on JsonSchemaType

github-actions · 2022-12-23T21:49:55Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.34`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.34`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.14`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.14`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.18`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.18`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.16`	✅	✅
`source-db2-strict-encrypt`	`0.1.16`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.0`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`0.4.26`	✅	✅
`source-mssql-strict-encrypt`	`0.4.26`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`1.0.18`	✅	✅
`source-mysql-strict-encrypt`	`1.0.18`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.21`	✅	✅
`source-oracle-strict-encrypt`	`0.3.21`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`1.0.34`	✅	✅
`source-postgres-strict-encrypt`	`1.0.34`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.15`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.27`	✅	✅
`source-tidb`	`0.2.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

akashkulk · 2022-12-24T00:08:54Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3768931556
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3768931556
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================= 13 passed, 6 skipped, 21 warnings in 16.78s ==================

akashkulk · 2022-12-24T00:09:03Z

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3768932089
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3768932089
No Python unittests run

Build Passed

Test summary info:

All Passed

rodireich · 2022-12-27T18:56:14Z

...rc/test-integration/java/io/airbyte/integrations/source/oracle/OracleSourceDatatypeTest.java

+            /*
+             * because normalization expects a plain integer strings whereas `Math.pow(10, 125)` returns a
+             * scientific notation
+             */


Couldn't have put it better myself ;-)

akashkulk · 2023-01-03T07:31:19Z

/test connector=connectors/source-postgres

akashkulk · 2023-01-03T07:33:25Z

/test connector=connectors/source-postgres-strict-encrypt

🕑 connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3827672088
✅ connectors/source-postgres-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3827672088
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2023-01-03T07:41:14Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3827706289
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3827706289
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================= 13 passed, 6 skipped, 21 warnings in 15.30s ==================

akashkulk · 2023-01-03T07:41:23Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3827706851
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3827706851
No Python unittests run

Build Passed

Test summary info:

All Passed

akashkulk · 2023-01-03T07:43:29Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3827715373
❌ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/3827715373
🐛 https://gradle.com/s/imsarg625rwr4

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2023-01-03T07:44:32Z

/test connector=connectors/source-mongodb

🕑 connectors/source-mongodb https://github.com/airbytehq/airbyte/actions/runs/3827719599
❌ connectors/source-mongodb https://github.com/airbytehq/airbyte/actions/runs/3827719599
🐛 https://gradle.com/s/pbsymrxpdjm42

Build Failed

Test summary info:

Could not find result summary

akashkulk · 2023-01-03T08:41:05Z

/test connector=connectors/source-postgres

akashkulk · 2023-01-03T08:41:34Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/3828030815
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/3828030815
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1603    336    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================= 13 passed, 6 skipped, 21 warnings in 17.48s ==================

akashkulk · 2023-01-03T09:08:11Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3828189226
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/3828189226
🐛 https://gradle.com/s/4l24wi7rpj3rc

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============= 2 failed, 15 passed, 1 skipped, 10 errors in 31.69s ==============

VitaliiMaltsev · 2023-01-16T13:35:19Z

/test connector=connectors/source-cockroachdb

VitaliiMaltsev · 2023-01-16T13:35:40Z

/test connector=connectors/source-clickhouse

Refactor SourceOperations class

e0b5e5f

akashkulk temporarily deployed to more-secrets December 23, 2022 21:49 — with GitHub Actions Inactive

akashkulk temporarily deployed to more-secrets December 23, 2022 21:50 — with GitHub Actions Inactive

akashkulk changed the title ~~Refactor SourceOperations class~~ Code cleanup in SourceOperations Dec 24, 2022

More cleanup

b0a1aae

octavia-squidington-iv added connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/source/db2-strict-encrypt connectors/source/oracle connectors/source/oracle-strict-encrypt labels Dec 24, 2022

akashkulk temporarily deployed to more-secrets December 24, 2022 00:10 — with GitHub Actions Inactive

akashkulk temporarily deployed to more-secrets December 24, 2022 00:11 — with GitHub Actions Inactive

akashkulk marked this pull request as ready for review December 27, 2022 17:36

rodireich approved these changes Dec 27, 2022

View reviewed changes

Merge branch 'master' into comment

4459220

akashkulk temporarily deployed to more-secrets January 3, 2023 07:31 — with GitHub Actions Inactive

akashkulk temporarily deployed to more-secrets January 3, 2023 07:32 — with GitHub Actions Inactive

akashkulk temporarily deployed to more-secrets January 3, 2023 08:18 — with GitHub Actions Inactive

Formatting

60008b3

akashkulk temporarily deployed to more-secrets January 3, 2023 08:25 — with GitHub Actions Inactive

akashkulk temporarily deployed to more-secrets January 3, 2023 08:26 — with GitHub Actions Inactive

Merge branch 'master' into comment

e7a726a

akashkulk temporarily deployed to more-secrets January 3, 2023 10:24 — with GitHub Actions Inactive

akashkulk merged commit bac789e into master Jan 3, 2023

akashkulk deleted the comment branch January 3, 2023 11:09

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21092

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21121

Closed

github-actions bot mentioned this pull request Jan 6, 2023

Bump helm chart version reference to 0.43.15 #21125

Closed

This was referenced Jan 6, 2023

Bump Airbyte version from 0.40.26 to 0.40.27 #21130

Closed

Bump Airbyte version from 0.40.26 to 0.40.27 #21135

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Code cleanup in `SourceOperations` #20874

Code cleanup in `SourceOperations` #20874

akashkulk commented Dec 23, 2022 •

edited

Loading

github-actions bot commented Dec 23, 2022 •

edited

Loading

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Dec 24, 2022 •

edited by github-actions bot

Loading

akashkulk commented Dec 24, 2022 •

edited by github-actions bot

Loading

rodireich Dec 27, 2022

akashkulk commented Jan 3, 2023

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Jan 16, 2023

VitaliiMaltsev commented Jan 16, 2023

Code cleanup in SourceOperations #20874

Code cleanup in SourceOperations #20874

Conversation

akashkulk commented Dec 23, 2022 • edited Loading

github-actions bot commented Dec 23, 2022 • edited Loading

Affected Connector Report

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Dec 24, 2022 • edited by github-actions bot Loading

Build Passed

akashkulk commented Dec 24, 2022 • edited by github-actions bot Loading

Build Passed

rodireich Dec 27, 2022

Choose a reason for hiding this comment

akashkulk commented Jan 3, 2023

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Failed

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Failed

akashkulk commented Jan 3, 2023

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Jan 3, 2023 • edited by github-actions bot Loading

Build Failed

VitaliiMaltsev commented Jan 16, 2023

VitaliiMaltsev commented Jan 16, 2023

Code cleanup in `SourceOperations` #20874

Code cleanup in `SourceOperations` #20874

akashkulk commented Dec 23, 2022 •

edited

Loading

github-actions bot commented Dec 23, 2022 •

edited

Loading

akashkulk commented Dec 24, 2022 •

edited by github-actions bot

Loading

akashkulk commented Dec 24, 2022 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading

akashkulk commented Jan 3, 2023 •

edited by github-actions bot

Loading