postgres-source-cdc: handle null values for array data types #21003

subodh1810 · 2023-01-04T06:30:15Z

issue : #20930

github-actions · 2023-01-04T06:32:03Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (6)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.34`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.34`	🔵 (ignored)	🔵 (ignored)
`source-mssql`	`0.4.26`	✅	✅
`source-mysql`	`1.0.18`	✅	✅
`source-postgres`	`1.0.36`	✅	✅
`source-postgres-strict-encrypt`	`1.0.36`	🔵 (ignored)	🔵 (ignored)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

subodh1810 · 2023-01-04T07:16:21Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3835978251
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3835978251
🐛 https://gradle.com/s/5ioblobdrp6nq

Build Failed

Test summary info:

Could not find result summary

subodh1810 · 2023-01-04T10:11:00Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3837050238
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3837050238
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1606    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
================= 13 passed, 6 skipped, 21 warnings in 15.28s ==================

subodh1810 · 2023-01-06T10:24:31Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3854490911
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3854490911
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       141      5    96%   87, 93, 239, 243-244
	 source_acceptance_test/conftest.py                     211     95    55%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-285, 293-306, 311-317, 324-335, 342-358
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              402    115    71%   53, 58, 93-104, 109-116, 120-121, 125-126, 308, 346-363, 376-387, 391-396, 402, 435-440, 478-485, 528-530, 533, 598-606, 618-621, 626, 682-683, 689, 692, 728-738, 751-776
	 source_acceptance_test/tests/test_incremental.py       160     14    91%   56-63, 68-81, 244
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1609    339    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:94: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_core.py:377: The previous and actual discovered catalogs are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/tests/test_incremental.py:22: `future_state` has a bypass reason, skipping.
================== 54 passed, 6 skipped in 717.69s (0:11:57) ===================

subodh1810 · 2023-01-11T05:32:58Z

/publish connector=connectors/source-postgres-strict-encrypt

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-01-11T05:33:06Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/3890004234

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

subodh1810 · 2023-01-11T12:22:49Z

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3892749809

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* postgres-source-cdc:handle null values for array data types * fix test * upgrade version * auto-bump connector version Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

postgres-source-cdc:handle null values for array data types

8c52a80

subodh1810 requested review from rodireich and akashkulk January 4, 2023 06:30

subodh1810 self-assigned this Jan 4, 2023

rodireich approved these changes Jan 4, 2023

View reviewed changes

subodh1810 added 2 commits January 4, 2023 14:34

Merge branch 'master' into handle_npe_array_type

6792731

fix test

b527d2c

subodh1810 requested a review from rodireich January 4, 2023 10:11

rodireich approved these changes Jan 4, 2023

View reviewed changes

Merge branch 'master' into handle_npe_array_type

2179705

subodh1810 added 2 commits January 11, 2023 10:55

Merge branch 'master' into handle_npe_array_type

8e52297

upgrade version

6969d86

subodh1810 requested a review from a team as a code owner January 11, 2023 05:30

octavia-squidington-iv added connectors/source/postgres connectors/source/postgres-strict-encrypt area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Jan 11, 2023

subodh1810 temporarily deployed to more-secrets January 11, 2023 05:32 — with GitHub Actions Inactive

subodh1810 temporarily deployed to more-secrets January 11, 2023 05:33 — with GitHub Actions Inactive

auto-bump connector version

b3ef4da

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2023 06:39 — with GitHub Actions Inactive

octavia-squidington-iii temporarily deployed to more-secrets January 11, 2023 06:40 — with GitHub Actions Inactive

subodh1810 merged commit 8dffde0 into master Jan 11, 2023

subodh1810 deleted the handle_npe_array_type branch January 11, 2023 18:11

octavia-squidington-iii mentioned this pull request Jan 12, 2023

Bump Airbyte version from 0.40.27 to 0.40.28 #21359

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

postgres-source-cdc: handle null values for array data types #21003

postgres-source-cdc: handle null values for array data types #21003

subodh1810 commented Jan 4, 2023

github-actions bot commented Jan 4, 2023 •

edited

Loading

✅ Sources (6)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 commented Jan 4, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 4, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 6, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading

postgres-source-cdc: handle null values for array data types #21003

postgres-source-cdc: handle null values for array data types #21003

Conversation

subodh1810 commented Jan 4, 2023

github-actions bot commented Jan 4, 2023 • edited Loading

Affected Connector Report

✅ Sources (6)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

subodh1810 commented Jan 4, 2023 • edited by github-actions bot Loading

Build Failed

subodh1810 commented Jan 4, 2023 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Jan 6, 2023 • edited by github-actions bot Loading

Build Passed

subodh1810 commented Jan 11, 2023 • edited by github-actions bot Loading

subodh1810 commented Jan 11, 2023 • edited by github-actions bot Loading

subodh1810 commented Jan 11, 2023 • edited by github-actions bot Loading

github-actions bot commented Jan 4, 2023 •

edited

Loading

subodh1810 commented Jan 4, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 4, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 6, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading

subodh1810 commented Jan 11, 2023 •

edited by github-actions bot

Loading