Implement Debezium heartbeats for source-postgres #19004

rodireich · 2022-11-05T04:27:31Z

What

This is adding heartbeat functionality to Debezium and implements it for source postgres CDC.

How

Processing incoming heartbeat events which add more predictability to our sync and prevent us from wrapping up too soon in some cases. See discussion in #15040.
This while keeping existing functionality for other CDC connectors - MySQL and MSSql which will have heartbeat added later on.

🚨 User Impact 🚨

New logic should improve our syncing in some of the harder cases. e.g large and busy database out of which we only sync a small amount of tables.

github-actions · 2022-11-05T04:28:27Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector version
Add changelog
Publish the new version

⚠ Sources (7)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.17`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.17`	✅	⚠ (not in seed)
`source-mssql`	`0.4.25`	✅	✅
`source-mysql`	`1.0.13`	✅	✅
`source-mysql-strict-encrypt`	`1.0.13`	✅	⚠ (not in seed)
`source-postgres`	`1.0.25`	✅	✅
`source-postgres-strict-encrypt`	`1.0.25`	✅	⚠ (not in seed)

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

rodireich · 2022-11-05T04:29:34Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3398677111
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3398677111
No Python unittests run

Build Passed

Test summary info:

All Passed

…mpilation.

rodireich · 2022-11-07T06:30:03Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3408077843
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3408077843
No Python unittests run

Build Passed

Test summary info:

All Passed

rodireich · 2022-11-07T06:30:12Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3408078606
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/3408078606
No Python unittests run

Build Passed

Test summary info:

All Passed

rodireich · 2022-11-07T06:30:21Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/3408078869
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/3408078869
No Python unittests run

Build Passed

Test summary info:

All Passed

rodireich · 2022-11-08T05:44:24Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3416728030
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3416728030
No Python unittests run

Build Passed

Test summary info:

All Passed

…gres-cdc

alafanechere · 2022-11-16T14:06:46Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3480058606
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3480058606
🐛 https://gradle.com/s/4xgpmcrlnhuic

Build Failed

Test summary info:

Could not find result summary

alafanechere · 2022-11-16T14:11:55Z

@rodireich the acceptance-test-config.yml format was not corresponding to the newest one. (it was like a mix between the legacy one that is still supported and the new one 😄 ) . You'll find more details here.

alafanechere · 2022-11-16T15:04:47Z

I'm under the impression that the preivous /test failed before SAT . Re-running in case it's flaky.

alafanechere · 2022-11-16T16:22:34Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3481159996
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3481159996
🐛 https://gradle.com/s/3axcxossriv5o

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestSpec::test_enum_usage[inputs0] - TypeError: list ind...
FAILED test_core.py::TestSpec::test_backward_compatibility[inputs0] - hypothe...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: Pending.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: Pending.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: Pending.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: Pending.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: Pending.
============= 2 failed, 12 passed, 5 skipped in 175.07s (0:02:55) ==============

rodireich · 2022-11-16T21:49:38Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3483393867
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3483393867
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       139      5    96%   87, 93, 235, 239-240
	 source_acceptance_test/conftest.py                     196     92    53%   35, 41-43, 48, 54, 60, 66, 72-74, 93, 98-100, 106-108, 114-115, 120-121, 126, 132, 141-150, 156-161, 176, 200, 231, 237, 243-248, 256-261, 269-282, 287-293, 300-311, 318-334
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 37      2    95%   57-58
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1563    349    78%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 27.25s ==================

…gres-cdc

rodireich · 2022-11-16T23:23:55Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/3483948735

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…gres-cdc

rodireich · 2022-11-17T00:20:26Z

/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2022-11-17T11:04:04Z

@rodireich I'm under the impression the cause of this failure is not the config format change but rather a flaky behavior on backward compatibility checks. The test strictness level does not change the behavior of spec testing and the same tests were run in your latest successful attempt.
I'm sorry for this flaky behavior and opened an issue to try to fix it if's occurring more often .

* Initial working commit * Code sanity. Provide no-on implementation to mysql, MSSql to allow compilation. * Update test * sanity * sanity * sanity * sanity * sanity * changes per review comments * Make heartbeat change waittime configurable. * Trying to bypass test strictness test * Trying to bypass test strictness test * Trying to bypass test strictness test * fix acceptance test config format * add missing SAT test in config * revert back changes in acceptance-test-config.yml * Version and notes * auto-bump connector version Co-authored-by: alafanechere <augustin.lafanechere@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

Initial working commit

6f1eb6f

rodireich linked an issue Nov 5, 2022 that may be closed by this pull request

Implement heartbeats for source-postgres CDC #18987

Closed

github-actions bot added the area/connectors Connector related issues label Nov 5, 2022

octavia-squidington-iv added connectors/source/mssql connectors/source/mysql connectors/source/postgres labels Nov 5, 2022

rodireich temporarily deployed to more-secrets November 5, 2022 04:28 Inactive

Code sanity. Provide no-on implementation to mysql, MSSql to allow co…

f7429de

…mpilation.

rodireich temporarily deployed to more-secrets November 7, 2022 06:30 Inactive

Update test

dd65a0b

rodireich temporarily deployed to more-secrets November 8, 2022 05:37 Inactive

sanity

0bb5b7b

rodireich temporarily deployed to more-secrets November 8, 2022 05:46 Inactive

rodireich changed the title ~~Initial working commit~~ Implement Debezium heartbeats for source-postgres Nov 8, 2022

sanity

8146400

rodireich temporarily deployed to more-secrets November 8, 2022 23:40 Inactive

rodireich marked this pull request as ready for review November 9, 2022 06:48

rodireich requested a review from a team as a code owner November 9, 2022 06:48

Merge branch 'master' into 18987-implement-heartbeats-for-source-post…

5b8f0c0

…gres-cdc

rodireich temporarily deployed to more-secrets November 9, 2022 16:49 Inactive

Merge branch 'master' into 18987-implement-heartbeats-for-source-post…

ef030d2

…gres-cdc

rodireich temporarily deployed to more-secrets November 9, 2022 18:11 Inactive

rodireich assigned subodh1810 Nov 10, 2022

rodireich requested a review from VitaliiMaltsev November 10, 2022 00:17

alafanechere temporarily deployed to more-secrets November 16, 2022 14:08 Inactive

add missing SAT test in config

f7fd13e

alafanechere temporarily deployed to more-secrets November 16, 2022 16:24 Inactive

rodireich assigned rodireich and unassigned subodh1810 Nov 16, 2022

revert back changes in acceptance-test-config.yml

fd2d619

Merge branch 'master' into 18987-implement-heartbeats-for-source-post…

3d33bce

…gres-cdc

rodireich temporarily deployed to more-secrets November 16, 2022 21:51 Inactive

Version and notes

9919e53

octavia-squidington-iv added connectors/source/postgres-strict-encrypt area/documentation Improvements or additions to documentation labels Nov 16, 2022

rodireich temporarily deployed to more-secrets November 16, 2022 23:22 Inactive

Merge branch 'master' into 18987-implement-heartbeats-for-source-post…

10ac762

…gres-cdc

rodireich temporarily deployed to more-secrets November 16, 2022 23:26 Inactive

auto-bump connector version

65f8764

octavia-squidington-iii temporarily deployed to more-secrets November 17, 2022 00:10 Inactive

rodireich merged commit c739c4c into master Nov 17, 2022

rodireich deleted the 18987-implement-heartbeats-for-source-postgres-cdc branch November 17, 2022 00:20

alafanechere mentioned this pull request Nov 17, 2022

Source Postgres: enable high test strictness level in SAT #19060

Closed

github-actions bot mentioned this pull request Nov 17, 2022

Bump helm chart version reference to 0.40.44 #19546

Merged

octavia-squidington-iii mentioned this pull request Nov 18, 2022

Bump Airbyte version from 0.40.18 to 0.40.19 #19579

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Debezium heartbeats for source-postgres #19004

Implement Debezium heartbeats for source-postgres #19004

rodireich commented Nov 5, 2022 •

edited

Loading

github-actions bot commented Nov 5, 2022 •

edited

Loading

⚠ Sources (7)

✅ Destinations (0)

Actionable Items

rodireich commented Nov 5, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 8, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 16, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 16, 2022

alafanechere commented Nov 16, 2022

alafanechere commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 17, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 17, 2022

Implement Debezium heartbeats for source-postgres #19004

Implement Debezium heartbeats for source-postgres #19004

Conversation

rodireich commented Nov 5, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

github-actions bot commented Nov 5, 2022 • edited Loading

Affected Connector Report

⚠ Sources (7)

✅ Destinations (0)

Actionable Items

rodireich commented Nov 5, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Nov 7, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Nov 7, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Nov 7, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Nov 8, 2022 • edited by github-actions bot Loading

Build Passed

alafanechere commented Nov 16, 2022 • edited by github-actions bot Loading

Build Failed

alafanechere commented Nov 16, 2022

alafanechere commented Nov 16, 2022

alafanechere commented Nov 16, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Nov 16, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Nov 16, 2022 • edited by github-actions bot Loading

rodireich commented Nov 17, 2022 • edited by github-actions bot Loading

alafanechere commented Nov 17, 2022

rodireich commented Nov 5, 2022 •

edited

Loading

github-actions bot commented Nov 5, 2022 •

edited

Loading

rodireich commented Nov 5, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 7, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 8, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 16, 2022 •

edited by github-actions bot

Loading

alafanechere commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 16, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 17, 2022 •

edited by github-actions bot

Loading