[xCluster + Tablet Splitting] Support splits on the producer #10874
Assignees
Labels
area/cdc
Change Data Capture
area/docdb
YugabyteDB core features
kind/bug
This issue is a bug
priority/medium
Medium priority issue
Comments
17 tasks
hulien22
added a commit
that referenced
this issue
Jan 22, 2022
Summary: Adding support for sending SPLIT_OP records from the producer to the consumer. Upon receiving a SPLIT_OP record, the consumer updates its state and creates two new pollers for the new children tablets. To ensure proper processing of records, the consumer will now process all writes it has received before the SPLIT_OP, and once those writes are complete, it will process the split (this is to ensure that we have processed all of the previous records for the parent tablet before we delete the poller for that parent). This change is done in the `TwoDCOutputClient` under `ProcessChangesStartingFromIndex`. Processing the split involves sending a `UpdateConsumerOnProducerSplit` request to the catalog manager. This call will remove the parent tablet from the tablet mapping in the stream's metadata, and replace it with the children. In the future, we can improve the placement of the pollers for these children. When processing the split, we still follow the at-least-once semantics we have been using for replication. That means that if we receive the same SPLIT_OP (which currently happens since the cdc_state checkpoint is currently just set to 0.0, so we will reprocess any old SPLIT_OPs in the child's logs), we will recognize that we have already processed it, and continue. On the producer side, also adding in support for handling GetChanges requests to new children tablets (simply requires adding in new entries to `tablet_checkpoints_`). Test Plan: ``` ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterTabletSplitITest.SplittingWithXClusterReplicationOnProducer ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterTabletSplitITest.MultipleSplitsDuringPausedReplication ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterTabletSplitITest.MultipleSplitsInSequence ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterTabletSplitITest.SplittingOnProducerAndConsumer ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterBootstrapTabletSplitITest.BootstrapWithSplits ybd --cxx_test xcluster-tablet-split-itest --gtest-filter XClusterTabletSplitITest.ConsumerClusterFailureWhenProcessingSplitOp ``` Reviewers: rahuldesirazu, nicolas Reviewed By: nicolas Subscribers: ybase, bogdan Differential Revision: https://phabricator.dev.yugabyte.com/D14371
yugabyte-ci
added
kind/bug
|
Closed by 9f1497a |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Jira Link: DB-940
Support a tablet split that occurs on the producer. This needs to send the split_op to the consumer, which then needs to create new pollers for the new children tablets.
The text was updated successfully, but these errors were encountered: