auto discover and switch between DM-master addresses for dmctl

[csuzhangxc]

Feature Request

Is your feature request related to a problem? Please describe:

now, dmctl can only specify a DM-master address statically.

if the specified DM-master is down, then we need to specify another DM-master's address manually.

Describe the feature you'd like:

discover and switch between DM-master addresses for dmctl automatically.

Describe alternatives you've considered:

Teachability, Documentation, Adoption, Migration Strategy:

[lance6716]

seems PD doesn't forward between etcd members, pd-ctl have all pd's address and try one by one

https://docs.pingcap.com/zh/tidb/stable/pd-control#--pd---u

https://github.com/tikv/pd/blob/6b0f493ec47829b5a8242a5af1634216159dd229/tools/pd-ctl/pdctl/command/global.go#L81-L82

[csuzhangxc]

Can dmctl fetch all DM-master's address periodically and try to use them one by one?

[lance6716]

Can dmctl fetch all DM-master's address periodically and try to use them one by one?

currently we could query list-member after a success command, save them in hidden cache file or something, and use it when some specific query fail. this "try next master" maybe duplicated with internal forwarding, and since internal forwarding caused many problems, we might disable it in future?

[csuzhangxc]

HTTP API may still need internal forwarding.

[lance6716]

closed by #1333