[WIP] Add optimistic sharding mode

[kennytm]

WIP! Borrowing the CI for testing.

What problem does this PR solve?

Add a simple optimistic DDL lock mode. Enable with dmctl set-ddl-lock-mode optimistic.

What is changed and how it works?

Check List

Tests

• Unit test

Code changes

Side effects

Related changes

• Need to update the documentation
• Need to be included in the release note

[kennytm]

ok i'm out of idea how the optimistic lock should fit into the existing system.

current pessimistic-only system is roughly like:

// master 1
go {
    for {
        task := <-infoCh 
        lock := FindLock(task)
        synced := lock.TrySync(task)
        if synced {
            opCh <- NewOperation(lock)
        }
    }
}    

// master 2
go {
    for {
        op := <-opCh 
        lock := FindLock(op)
        for _, source := range lock.Sources {
            if source == lock.Owner {
                clientCh <- "execute " + lock.DDLs
            } else {
                clientCh <- "skip " + lock.DDLs 
            }
        }
        RemoveLock(op)
    }
}

// client 
go {
    ...
    infoCh <- DDLs 
    action := <-clientCh  // (reader responsible for deleting the etcd keys??)
    ...
}

supporting optimistic-lock should be simply changing:

// master 1
go {
    for {
        task := <-infoCh 
        lock := FindLock(task)
        newDDLs, err := lock.TrySync(task)       // <-- changed 
        if err != nil {                          
            opCh <- NewOperation(task, newDDLs)
        }
    }
}    

// master 2
go {
    for {
        op := <-opCh 
        lock := FindLock(op)
        for _, source := range lock.Sources {
            if source == lock.Owner {
                clientCh <- "execute " + op.DDLs // <-- changed
            } else {
                clientCh <- "skip"               // <-- changed 
            }
        }
        if lock.Synced() {                       // <-- changed 
            RemoveLock(op)
        }
    }
}

// client 
go {
    ...
    infoCh <- DDLs 
    action := <-clientCh
    ...
}

but i can't get the client to accept the modified DDLs 😕

[csuzhangxc]

in #522 I change only the leader of the master to coordinate shard DDL in pessimistic mode.

In pessimistic mode, the Info should be deleted by the worker, the Operation should be deleted by the master, you can see pkg/pessimism/doc.go.

[csuzhangxc]

Do we need to set a different mode for each migration task?

If so, maybe we can add a config item in task.yaml?

[csuzhangxc]

set ddls = nil for err == nil && cmpRes == 0 (l.synced = true)?

[csuzhangxc]

implemented in #568.