[DocDB] [xCluster] Replication is not working if user upgrades universes to master branch #11729
Assignees
Labels
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
xCluster
Label for xCluster related issues/improvements
Comments
vaibhav-yb
added a commit
that referenced
this issue
Mar 23, 2022
…rades to a branch with CDCSDK code changes Summary: With the changes for CDCSDK, we have separate `source_type` values i.e. `XCLUSTER` for xCluster replication and `CDCSDK` for the new changes. Similarly there is another option i.e. `checkpoint_type` which can have `IMPLICIT` and `EXPLICIT` values. If a stream for replication has been created before upgrading, it was unable to continue replication after upgrading to the latest version since the `source_type` and `checkpoint_type` options were missing from it as it has only been introduced with the CDCSDK changes only. Test Plan: * Manually tested with a custom build on dev portal Reviewers: sergei, jhe, mkantimath, skumar, rahuldesirazu Reviewed By: skumar, rahuldesirazu Subscribers: rahuldesirazu, iamoncar, sdash, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D15989
suranjan
pushed a commit
that referenced
this issue
Mar 29, 2022
…Subscriber Summary: Original commits: - d294abf / D13836 - 6b15b16 / D15860 - cf5fead / D16057 - 5408e30 / D15989 - 53afee99e7cfe912fa7ec579ce470e999701a074 / D16176 Github Master Ticket: #9019 Design DocumentL https://docs.google.com/document/d/1_xZqU5UgzCu1W--kci3ajU7_iYXXHMQvudmybDI-Xsk/edit Functional Spec: https://docs.google.com/document/u/2/d/1nHuzHQ-qYVPbKi2dqo_drzSXMq00h7w5oi0JDf0GD1U/edit#heading=h.jmqfs7jgvvg8 This is the client-side change that exposes some APIs to be consumed by CDC consumers. Currently, these APIs are not public and are to be consumed by our Debezium connector. For testing purposes, we have written a console subscriber for testing purposes. [CDCSDK][#11679] Add missing license headers to Java files The following files have missing license headers: 1. `CreateCDCStreamRequest.java` 2. `GetCheckpointRequest.java` 3. `GetCheckpointResponse.java` 4. `GetDBStreamInfoRequest.java` 5. `GetDBStreamInfoResponse.java` 6. `SetCheckpointRequest.java` 7. `SetCheckpointResponse.java` This diff adds these missing headers. [#11779][CDCSDK] Add option to send a DDL record based on a flag value in GetChangesRequest Before this, the issue was that if for a stream ID, some data was consumed and a client comes up with the same stream ID and requests for changes, it will only receive the changes. Now the issue with this was with `Debezium` that when the connector was restarted, it directly received the changes without any DDL record, this DDL record was essential for Debezium since it was used to process the schema info for the columns in Debezium and in case it was not there, it lead to a `NullPointerException` on the client side, thus causing a connector crash effectively. [#11729][DocDB][xCluster] Fix for replication not working if user upgrades to a branch with CDCSDK code changes With the changes for CDCSDK, we have separate `source_type` values i.e. `XCLUSTER` for xCluster replication and `CDCSDK` for the new changes. Similarly there is another option i.e. `checkpoint_type` which can have `IMPLICIT` and `EXPLICIT` values. If a stream for replication has been created before upgrading, it was unable to continue replication after upgrading to the latest version since the `source_type` and `checkpoint_type` options were missing from it as it has only been introduced with the CDCSDK changes only. Test Fixes for 2.12 Test Plan: Jenkins: skip Unit tests in java for APIs and CDC behavior. We have done some long-running testing with applications. We have also run the YB-sample apps and enabled CDC on the table. Verified that all the events are received. Tested the complete CDC with Debezium pipeline with the specified change. Command to run test: `ybd --cxx-test integration-tests_cdcsdk_ysql-test --gtest_filter CDCSDKYsqlTest.TestNeedSchemaInfoFlag` * Manually tested with a custom build on dev portal Reviewers: nicolas, bogdan, ybase, rahuldesirazu, sdash, iamoncar, zyu, jhe, mkantimath, sergei Reviewed By: sergei Differential Revision: https://phabricator.dev.yugabyte.com/D16251
nathanhjli
pushed a commit
to nathanhjli/yugabyte-db
that referenced
this issue
Mar 31, 2022
…user upgrades to a branch with CDCSDK code changes Summary: With the changes for CDCSDK, we have separate `source_type` values i.e. `XCLUSTER` for xCluster replication and `CDCSDK` for the new changes. Similarly there is another option i.e. `checkpoint_type` which can have `IMPLICIT` and `EXPLICIT` values. If a stream for replication has been created before upgrading, it was unable to continue replication after upgrading to the latest version since the `source_type` and `checkpoint_type` options were missing from it as it has only been introduced with the CDCSDK changes only. Test Plan: * Manually tested with a custom build on dev portal Reviewers: sergei, jhe, mkantimath, skumar, rahuldesirazu Reviewed By: skumar, rahuldesirazu Subscribers: rahuldesirazu, iamoncar, sdash, ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D15989
yugabyte-ci
added
kind/bug
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-1105
Description
Steps:
Actual: Replication is not working after upgrading universes to master branch.
The text was updated successfully, but these errors were encountered: