Skip to content
This repository has been archived by the owner on Nov 24, 2023. It is now read-only.

shard DDL: rewrite shard DDL in pessimism mode for HA #456

Merged
merged 70 commits into from
Feb 11, 2020
Merged
Show file tree
Hide file tree
Changes from 67 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
ad3e066
fix all_mode integration_test
lichunzhu Jan 2, 2020
b3b3761
fix dmctl_advance & dmctl_basic integration_tests
lichunzhu Jan 3, 2020
eb6ec4c
initital commit for dmctl_command
lichunzhu Jan 3, 2020
7230952
fix full_mode test
lichunzhu Jan 3, 2020
1a865e4
refine http_apis integration_test
lichunzhu Jan 3, 2020
9a47eb0
fix import goroutine leak test
lichunzhu Jan 3, 2020
80e0269
fix incremental_mode integration_test
lichunzhu Jan 3, 2020
2400863
fix ha-dev bug
lichunzhu Jan 3, 2020
f337799
refine initial_unit integration_test
lichunzhu Jan 3, 2020
634d67b
fix load_interrupt integration_tests
lichunzhu Jan 3, 2020
1a807a4
fix online ddl integration_tests
lichunzhu Jan 3, 2020
18b63d1
fix worker start for all integration_tests
lichunzhu Jan 3, 2020
6bd6452
fix tests
lichunzhu Jan 5, 2020
efd038b
add operator_worker test
lichunzhu Jan 6, 2020
b9c7819
fix bug
lichunzhu Jan 6, 2020
060bbd4
wait more time in retry_cancel test
lichunzhu Jan 6, 2020
ce7819f
wait more time in incremental_mode test
lichunzhu Jan 6, 2020
03fcf74
wait more time in load_interrupt test
lichunzhu Jan 6, 2020
565c9e9
address comments
lichunzhu Jan 6, 2020
a59eb51
wait for time in relay_interrupt
lichunzhu Jan 6, 2020
ea0bf02
*: handle shard DDL info in etcd
csuzhangxc Jan 8, 2020
c471f84
*: refine pkg
csuzhangxc Jan 8, 2020
5a06721
*: add shard DDL lock
csuzhangxc Jan 8, 2020
0edab10
*: add `LockKeeper`
csuzhangxc Jan 8, 2020
7596e48
*: add `Operation` for etcd
csuzhangxc Jan 8, 2020
9b6dbcd
shardddl: add the doc about the coordinate of a shard DDL lock
csuzhangxc Jan 9, 2020
666d315
shardddl: add DELETE op for Info
csuzhangxc Jan 9, 2020
385ba93
shardddl: add more operations for Operation
csuzhangxc Jan 9, 2020
f1b02f1
shardddl: add `PutOperationDeleteInfo`; refine test suite
csuzhangxc Jan 9, 2020
af586a2
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 11, 2020
628c966
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 13, 2020
9e0c598
shardddl: add `IsSynced` for shard DDL lock
csuzhangxc Jan 13, 2020
57d7f2d
shardddl: return revision when getting all Info and Operation
csuzhangxc Jan 13, 2020
1ef5670
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 15, 2020
c844cc5
shardddl: refine code and doc.
csuzhangxc Jan 15, 2020
914b22d
shardddl: add the skeleton of shard DDL handler in dm-master
csuzhangxc Jan 15, 2020
296c1b4
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 16, 2020
5351e65
shardddl: add and update API
csuzhangxc Jan 17, 2020
092c884
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 17, 2020
10ebe75
shardddl: add some logic for shard ddl lock resolving
csuzhangxc Jan 20, 2020
2a29b47
shardddl: refine code and deleting previous resolved lock
csuzhangxc Jan 20, 2020
ced26bc
shardddl: refine API, fix sources for the task
csuzhangxc Jan 20, 2020
1156097
shardddl: workaround one of the weak txn support of etcd
csuzhangxc Jan 21, 2020
70df6b9
shardddl: add some simple test case for dm-master
csuzhangxc Jan 21, 2020
cc31ff0
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Jan 21, 2020
f378fa9
shardddl: fix lock handle after restarted; add test cases
csuzhangxc Feb 3, 2020
7fdbc8d
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Feb 3, 2020
71b6b9a
dm-master: use new pessimist to handle shard DDL
csuzhangxc Feb 4, 2020
678b600
Merge remote-tracking branch 'remotes/origin/ha-dev' into ha-shard-ddl
csuzhangxc Feb 4, 2020
14da773
shardddl: add pessimist for syncer
csuzhangxc Feb 6, 2020
ba1c4bb
shardddl: record pending shard DDL info in pessimist
csuzhangxc Feb 6, 2020
ae2ab86
shardddl: record pending shard DDL lock operation in pessimist
csuzhangxc Feb 6, 2020
fb8d9fd
*: replace syner with new shard DDL implementation
csuzhangxc Feb 7, 2020
20ce591
*: fix dm-syncer
csuzhangxc Feb 7, 2020
dc4ad9c
*: remove unused protobuf message and service
csuzhangxc Feb 7, 2020
dc57d1d
*: remove tracer in syncer and tests
csuzhangxc Feb 7, 2020
fe43031
tests: fix safe_mode case
csuzhangxc Feb 7, 2020
461e686
shardddl: fix test
csuzhangxc Feb 7, 2020
c701422
tests: fix sequence_safe_mode case
csuzhangxc Feb 7, 2020
39f8b18
*: fix shard handle for `CREATE DATABASE/TABLE`
csuzhangxc Feb 8, 2020
14ef2f8
tests: fix load_interrupt case
csuzhangxc Feb 8, 2020
e12872f
tests: try fix load_interrupt case
csuzhangxc Feb 8, 2020
53d5262
tests: try fix load_interrupt case
csuzhangxc Feb 8, 2020
e78a4f3
tests: try fix all_mode case
csuzhangxc Feb 8, 2020
1c2df9e
tests: try fix incremental_mode case
csuzhangxc Feb 8, 2020
e581f14
tests: try fix ha case
csuzhangxc Feb 8, 2020
3ae58b6
*: address comments
csuzhangxc Feb 10, 2020
0f841ca
Update dm/master/shardddl/pessimist.go
csuzhangxc Feb 11, 2020
1ac1871
sharddll: debug CI
csuzhangxc Feb 11, 2020
e2972a2
*: address comments
csuzhangxc Feb 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion cmd/dm-syncer/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ func main() {
log.L().Info("", zap.Stringer("dm-syncer conf", conf))
})

sync := syncer.NewSyncer(conf)
sync := syncer.NewSyncer(conf, nil) // do not support shard DDL for singleton syncer.
sc := make(chan os.Signal, 1)
signal.Notify(sc, syscall.SIGHUP, syscall.SIGINT, syscall.SIGTERM, syscall.SIGQUIT)

Expand Down
7 changes: 7 additions & 0 deletions dm/common/common.go
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,13 @@ var (
// UpstreamSubTaskKeyAdapter used to store SubTask which are subscribing data from MySQL source.
// k/v: Encode(source-id, task-name) -> SubTaskConfig
UpstreamSubTaskKeyAdapter KeyAdapter = keyHexEncoderDecoder("/dm-master/upstream/subtask/")

// ShardDDLPessimismInfoKeyAdapter used to store shard DDL info in pessimistic model.
// k/v: Encode(task-name, source-id) -> shard DDL info
ShardDDLPessimismInfoKeyAdapter KeyAdapter = keyHexEncoderDecoder("/dm-master/shardddl-pessimism/info/")
// ShardDDLPessimismOperationKeyAdapter used to store shard DDL lock in pessimistic model.
// k/v: Encode(task-name, source-id) -> shard DDL operation
ShardDDLPessimismOperationKeyAdapter KeyAdapter = keyHexEncoderDecoder("/dm-master/shardddl-pessimism/operation/")
)

func keyAdapterKeysLen(s KeyAdapter) int {
Expand Down
Loading