[docdb] xCluster Roadmap Firehose

[nspiegelberg]

Jira Link: DB-1262

Hardening

Reliability

• HIGH PRI [M] [xCluster] Make xCluster replication APIs atomic  #10977
• [L] SDET Plan & Bug Fixing
  -- Process Resiliency (Master, TS for Producer & Consumer)
  -- Replication Falls Behind (Visibility, Stable Cluster)
  -- Setup & Drop Replication Resiliency & Speed
  -- Bootstrap -> Setup -> Replication Resiliency & Speed
  -- Transaction Status Tablet Server Dies
  -- Conflict Resolution
• [M] Pause/Resume Cluster [xCluster] Handle Pausing/Resuming Test Clusters with Replication #10084
  -- Log Retention Policy
• [M] Resiliency to Software Modifications (Drop Table)
  -- [M] [xCluster] : Fix CDC Stream Deletion Race Condition #9877
  -- [S] HIGH PRI xCluster: Add missing MarkUniverseReplicationFailed calls #10221

Performance

• [M] SDET Throughput Testing, Large Cluster & Heavy Volume
• [S/M] HIGH PRI Large Volume Processing
  -- Bootstrap Producer w/ 100 Tables [xCluster] Slow BootstrapProducer requests when using bootstrap_cdc_producer on many tables #10065
  -- Setup Consumer w/ 100 Tables [xCluster] Slow SetupReplication requests when using too many tables #10611
• [S] [xCluster] Locality-Aware Placement for M:N tablet mapping in setup_replication #10186
• [S] [xCluster]: Improve UpdateCheckpoint Speed & IOs by moving to Background Thread #9419
• LOW PRI. Add Incremental Backup Support for Snapshots

Atomicity

• HIGH PRI [L] [xCluster] Support Atomic and Ordered multi-shard Transactions  #10976 (https://docs.google.com/document/d/1LChFuwG3Nbpveoc4uBlDq4VKV-H1ZFCysJwbqj3Bq5s)

DDLs

• HIGH PRI [M] Tablet Split Support [docdb] Support tablet splitting + xcluster replication #10175
• HIGH PRI [L] Index Backfill [xCluster] Integration with index backfill #7613
• HIGH PRI [S] Drop / Truncate Handling [xCluster] Disallow DROP or TRUNCATE of Tables Under Replication #9421
  -- Disallow unless explicit override
• [L] Alter Table, Basic Columns
• LOW PRI. [M] Namespace Replication, Create Table Support

Usability

• [M] HIGH PRI [xCluster] Create tool or API for high-level replication stream validation #10980
• HIGH PRI [S] [xCluster] Create API to Wait for Replication Drain #10978
• HIGH PRI [S] [xCluster] Rename xCluster Commands with "xCluster" prefix #10981
• HIGH PRI [S] [xCluster] Add force option to delete_cdc_streams for streams that are a part of replication #10730
• [M] [xCluster] Support Database/Keyspace level replication #10984
• [S] HIGH PRI Wal Retention CLI Command New yb-admin command to set the wal retention time of a table #1931
• [S] HIGH PRI Backfill Needed API [DocDB][XCluster] Bootstrap Needed API #10645
• [S] Mixed TLS Support (TLS on one side, not on the other)
• [M] Consumer Side Metrics [xCluster] Add Consumer Metrics #9918
• [M] HIGH PRI Sanity Checks [docdb] XCluster Replication: Setup_universe_replication should perform more schema checks #9386
  -- ID Matching Setup_universe_replication should check IDs #10185
  -- UUID Validation
  -- New endpoint for checking/fixing metadata inconsistencies [xCluster] Add server side validation API to check/fix metadata inconsistencies #10733
• [S] Platform Integration
  -- Rename [DocDB][xCluster] Create Rename Replication Command #10563
  -- Full Rollback on Failure [DocDB][xCluster] Full Rollback of Failed Replication #10564
  -- LOW PRI. Delete only Consumer information [DocDB][xCluster] Option to Delete only Consumer & Give Producer Bootstrap IDs #10565
• [M] Kubernetes Support [xCluster] Make xCluster work in Kubernetes #2422
• LOW PRI. [M] CDC Follower Read Support
• LOW PRI. 1-Step Command for Bootstrap+Backup+Restore+Setup+Validate
• LOW PRI. Setup Replication for Namespace (1-step process)
• LOW PRI. GFLAG Synchronization between Consumer & Producer clusters
• LOW PRI. [M] 3+Way Replication Support
  -- LOW PRI. [S] Multi-Producer support on Consumer.
• LOW PRI. [M] Master Address Change [xcluster] Automatically update consumer view of producer master addresses in the event of quorum changes #9688

Testing

• [xCluster] Test PITR workflow #10982
• [xCluster] Test xCluster with RPC Compression #10985

Documentation (Deploy Public Docs)

• HIGH PRI. Document Bootstrap flow [Docs] Setting up xCluster Replication With Backup Restore #6870
• HIGH PRI. Document Inspect & Modify flow [Docs] [xCluster] Document Inspect & Modify flow  #10426
• HIGH PRI. Document Deploying xCluster in Mixed Mode TLS [docs] Describe the process for using xCluster replication with TLS #9160
• Document Existing ALTER Table flow. [docs] Document the recommend process for ALTERing tables when replication is enabled in 2dc setup #4571
• Document How to Handle Index Tables
• Update WAL Retention Process [Docs] Add/Update missing data in doc for xCluster  #10414
• Update TRUNCATE limitation [DOC] We should include truncate in limitations on xCluster replication #7846
• HIGH PRI. Document missing yb-admin commands [Docs] Document missing xcluster commands #11541, [docs][docdb] Document missing commands related to xcluster replication #9246
• [xCluster] Document Flow For Geo-partitioning #10983
• [Docs] Improve documentation for xcluster limitations #11532
• [xCluster][YB-ADMIN] Need latest changes' update on yb-admin utility commands for xCluster  #11192
• [DocDB] xcluster: Document how to setup in k8s when you have pod to pod connectivity #11559