Enhancement: improve MoveTables SwitchTraffic performance (getCopyState)

[maxenglander]

Feature Description

During MoveTables SwitchTraffic, there is a phase where wrangler queries the copystate of every VStream:

1. First SwitchTraffic calls canSwitch

   vitess/go/vt/wrangler/workflow.go

   Line 517 in b3f4202

   func (vrw *VReplicationWorkflow) canSwitch(keyspace, workflowName string) (reason string, err error) {

2. Which calls getStreams, which loads every VStream in the target keyspace

   vitess/go/vt/wrangler/vexec.go

   Lines 705 to 725 in b3f4202

   	query := `select
   	id,
   	source,
   	pos,
   	stop_pos,
   	max_replication_lag,
   	state,
   	db_name,
   	time_updated,
   	transaction_timestamp,
   	time_heartbeat,
   	time_throttled,
   	component_throttled,
   	message,
   	tags,
   	workflow_type,
   	workflow_sub_type,
   	defer_secondary_keys,
   	rows_copied
   	from _vt.vreplication`
   	results, err := wr.runVexec(ctx, workflow, keyspace, query, nil, false)

3. And then iterates over every VStream to get the replication status and copy state

   vitess/go/vt/wrangler/vexec.go

   Lines 736 to 743 in b3f4202

   	for primary, result := range results {
   	var rsrStatus []*ReplicationStatus
   	nqr := sqltypes.Proto3ToResult(result).Named()
   	if len(nqr.Rows) == 0 {
   	continue
   	}
   	for _, row := range nqr.Rows {
   	status, sk, err := wr.getReplicationStatusFromRow(ctx, row, primary)

4. Each of which involves a VReplicationExec call

   vitess/go/vt/wrangler/vexec.go

   Lines 905 to 909 in b3f4202

   	func (wr *Wrangler) getCopyState(ctx context.Context, tablet *topo.TabletInfo, id int32) ([]copyState, error) {
   	var cs []copyState
   	query := fmt.Sprintf("select table_name, lastpk from _vt.copy_state where vrepl_id = %d and id in (select max(id) from _vt.copy_state where vrepl_id = %d group by vrepl_id, table_name)",
   	id, id)
   	qr, err := wr.VReplicationExec(ctx, tablet.Alias, query)

When there are a lot of VStreams, this takes so long that the entire SwitchTraffic times out ends up in partially completed state.

Use Case(s)

Until recently, Vitess did not support changing the primary Vindex during MoveTables. Now it does. For Vitess clusters with lots of shards, say 128, a MoveTables operation to change a primary Vindex would involve 16384 VStreams. While each of the calls to get the copy state of those VStreams would be fast, in aggregate they can result in a SwitchTraffic timeout.