cdc:ignore timeout if snapshot is not complete #18959
Conversation
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
Affected Connector ReportNOTE
|
| Connector | Version | Changelog | Publish |
|---|---|---|---|
source-alloydb |
1.0.33 |
❌ (changelog missing) |
✅ |
source-alloydb-strict-encrypt |
1.0.33 |
🔵 (ignored) |
🔵 (ignored) |
source-mssql |
0.4.26 |
✅ | ✅ |
source-mssql-strict-encrypt |
0.4.26 |
🔵 (ignored) |
🔵 (ignored) |
source-mysql |
1.0.16 |
✅ | ✅ |
source-mysql-strict-encrypt |
1.0.16 |
🔵 (ignored) |
🔵 (ignored) |
source-postgres |
1.0.33 |
✅ | ✅ |
source-postgres-strict-encrypt |
1.0.33 |
🔵 (ignored) |
🔵 (ignored) |
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Destinations (0)
| Connector | Version | Changelog | Publish |
|---|
- See "Actionable Items" below for how to resolve warnings and errors.
✅ Other Modules (0)
Actionable Items
(click to expand)
| Category | Status | Actionable Item |
|---|---|---|
| Version | ❌ mismatch |
The version of the connector is different from its normal variant. Please bump the version of the connector. |
| ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
|
| Changelog | ⚠ doc not found |
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug. |
| ❌ changelog missing |
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog. | |
| Publish | ⚠ not in seed |
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug. |
| ❌ diff seed version |
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version. |
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
| LOGGER.info("Closing cause next is returned as null"); | ||
| requestClose(); | ||
| if ((!receivedFirstRecord || hasSnapshotFinished || maxInstanceOfNoRecordsFound >= 10) && !signalledClose) { | ||
| LOGGER.info("Closing cause next is returned as null"); |
There was a problem hiding this comment.
Can you add a comment or add a better log message here to reflect the new logic?
|
/test connector=connectors/source-mysql
Build PassedTest summary info: |
|
/test connector=connectors/source-postgres
Build PassedTest summary info: |
|
/test connector=connectors/source-mssql
Build FailedTest summary info: |
|
/test connector=connectors/source-mssql
Build PassedTest summary info: |
| @@ -80,9 +82,12 @@ protected ChangeEvent<String, String> computeNext() { | |||
| // if within the timeout, the consumer could not get a record, it is time to tell the producer to | |||
| // shutdown. | |||
| if (next == null) { | |||
| LOGGER.info("Closing cause next is returned as null"); | |||
| requestClose(); | |||
| if ((!receivedFirstRecord || hasSnapshotFinished || maxInstanceOfNoRecordsFound >= 10) && !signalledClose) { | |||
There was a problem hiding this comment.
In what case do we want to keep trying after queue gave us null?
There was a problem hiding this comment.
So the thing that happened with carecru's connection. Null was returned during snapshot cause Debezium took more time that 1 minute. We should have kept trying instead of shutting down
There was a problem hiding this comment.
This should hopefully be addressed by heartbeats
| @@ -46,6 +46,7 @@ public class DebeziumRecordIterator extends AbstractIterator<ChangeEvent<String, | |||
| private boolean receivedFirstRecord; | |||
| private boolean hasSnapshotFinished; | |||
| private boolean signalledClose; | |||
| private int maxInstanceOfNoRecordsFound; | |||
There was a problem hiding this comment.
Adding heartbeats make it unnecessary but since only postgres has heartbeats for now I'll make sure it doesn't collide with heartbeats
|
/publish connector=connectors/source-alloydb
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-alloydb-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-postgres
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-postgres-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mysql-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
|
/publish connector=connectors/source-mssql-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mysql
if you have connectors that successfully published but failed definition generation, follow step 4 here |
|
/publish connector=connectors/source-mssql
if you have connectors that successfully published but failed definition generation, follow step 4 here |
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
octavia-squidington-iii
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
octavia-squidington-iii
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
|
/publish connector=connectors/source-mssql-strict-encrypt
if you have connectors that successfully published but failed definition generation, follow step 4 here |
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
subodh1810
temporarily deployed
to
more-secrets
GitHub Actions
Inactive
Ref : https://airbytehq-team.slack.com/archives/C046N0TGRMX/p1667552437922099?thread_ts=1667481029.484859&cid=C046N0TGRMX
This change allows to ignore the
SUBSEQUENT_RECORD_WAIT_TIMEtimeout if a snapshot is going on.