Feature Request: make VDiff support multi-tenant migrations

[mcrauwel]

Feature Description

In #15403 we've implemented multi-tenant migrations and so far this is working very good. We now ran into an issue with vdiff. On one of our multi-tenant workflow we started a VDiff and that is reporting a bunch of extra rows in the target table (the rows from the other tenants outside of this workflow that have already been migrated).

Example:

Table sbtest1:
	State:            pending
	ProcessedRows:    13720191
	MatchingRows:     7720788
	ExtraRowsTarget:  5999403

The workflow definition contains the tenant_id that is being worked on, so vdiff should add a WHERE <tenant_id_field> = <tenant_id> filter clause to the target-side SELECT-query.

      "options": {
        "tenant_id": "123456",
        "strip_sharded_auto_increment": false,
        "shards": []
      }

Use Case(s)

Running vdiff on any multi-tenant migration will be affected by this.

[mcrauwel]

for completeness:

the tenant_id_field is defined in the VSchema:

  "multi_tenant_spec": {
    "tenant_id_column_name": "my_tenant_id_field",
    "tenant_id_column_type": "INT64"
  }

[mattlord]

Here's a test case:

git checkout main && make build

pushd examples/local

./101_initial_cluster.sh; mysql < ../common/insert_commerce_data.sql; ./201_customer_tablets.sh; ./202_move_tables.sh; ./203_switch_reads.sh; ./204_switch_writes.sh; ./205_clean_commerce.sh;  ./301_customer_sharded.sh; ./302_new_shards.sh; ./303_reshard.sh; ./304_switch_reads.sh; ./305_switch_writes.sh; ./306_down_shard_0.sh; ./307_delete_shard_0.sh

vtctldclient ApplyVSchema --vschema="$(vtctldclient GetVSchema customer | jq -r '.vindexes += { "unicode_loose_xxhash": {"type": "unicode_loose_xxhash"} }')" customer

vtctldclient ApplyVSchema --vschema="$(vtctldclient GetVSchema customer | jq -r '. += { "multi_tenant_spec": {"tenant_id_column_name": "sku", "tenant_id_column_type": "VARCHAR"} } ')" customer

vtctldclient ApplyVSchema --vschema="$(vtctldclient GetVSchema customer | jq -r '.tables += { "product": { "column_vindexes": [{"column": "sku", "name": "unicode_loose_xxhash"}]} }')" customer

vtctldclient MoveTables --workflow commerce2customer --target-keyspace customer create --source-keyspace commerce --tables "product" --tenant-id='SKU-1001'

sleep 10

mysql customer -e "insert into product values ('SKU-1002', 'stuff', 99)"

vtctldclient vdiff --target-keyspace customer --workflow commerce2customer create

sleep 5 

vtctldclient vdiff --target-keyspace customer --workflow commerce2customer show last

That will show the extra row on the target:

vtctldclient vdiff --target-keyspace customer --workflow commerce2customer show last
VDiff 84655431-4e9d-4854-9826-db01dea88fed scheduled on target shards, use show to view progress

VDiff Summary for customer.commerce2customer (84655431-4e9d-4854-9826-db01dea88fed)
State:        completed
RowsCompared: 2
HasMismatch:  true
StartedAt:    2024-07-01 18:16:45
CompletedAt:  2024-07-01 18:16:45

Table product:
	State:            completed
	ProcessedRows:    2
	MatchingRows:     1
	ExtraRowsTarget:  1

Use "--format=json" for more detailed output.

[deepthi]

@mcrauwel as noted in the linked PR, you can in fact run VDiff properly on a tenant-based migration by providing the correct --shards parameter.