Skip to content
This repository has been archived by the owner on Sep 30, 2024. It is now read-only.

support for force-master-failover command #199

Merged
merged 11 commits into from
Jun 11, 2017
Merged

Conversation

shlomi-noach
Copy link
Collaborator

This PR introduces the -c force-master-failover command.

This tells orchestrator to assume the master is dead, and that it should kick a full failover, by whatever means it finds necessary to promote a replica.

Usage:

orchestrator -c force-master-failover -alias mycluster

or

orchestrator -c force-master-failover -i some.instance.in.a.cluster

orchestrator will figure out what the master is, shut its eyes and make pretend it's dead, then kick a failover.

Contrast with force-master-takeover where the user specifies the replica which must get promoted (more control but potentially more complicated for the user).

Note: this PR extends #196

cc @github/database-infrastructure

@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 8, 2017 04:52 Active
@shlomi-noach
Copy link
Collaborator Author

Web interface will look like this:

screen shot 2017-06-08 at 08 28 41

@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 8, 2017 06:14 Active
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 8, 2017 06:59 Failure
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 8, 2017 07:01 Failure
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 8, 2017 07:01 Failure
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 8, 2017 07:03 Failure
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 8, 2017 07:03 Failure
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 8, 2017 07:04 Active
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 8, 2017 07:47 Active
@shlomi-noach shlomi-noach temporarily deployed to production June 8, 2017 07:49 Inactive
@shlomi-noach shlomi-noach deployed to production/hubot-slack June 8, 2017 13:09 Active
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 8, 2017 13:45 Active
@shlomi-noach shlomi-noach had a problem deploying to production/hubot-slack June 8, 2017 13:45 Failure
For masters: just fail over, or attempt to fail over onto specific instances. No more 'lite' recoveries. People really just want to fail over.
For intermediate masters: 'relocate-replicas' instead of match* commands; no 'lite' option
@jonahberquist
Copy link
Collaborator

Nice! That's a good interface update, too.

@shlomi-noach
Copy link
Collaborator Author

shlomi-noach commented Jun 10, 2017

Web interface simplified. For master, the "recover" options look like:

screen shot 2017-06-09 at 21 49 36

For intermediate master:

screen shot 2017-06-09 at 21 50 03

I completely removed the recover-lite options, as well as various "regroup" options. I'm htinking they're not really interesting, and are mostly confusing at time of failover.

@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 10, 2017 06:46 Active
@shlomi-noach shlomi-noach requested a deployment to production/hubot-slack June 10, 2017 06:46 Pending
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 11, 2017 05:23 Active
@shlomi-noach shlomi-noach deployed to production/hubot-slack June 11, 2017 05:26 Active
@shlomi-noach shlomi-noach had a problem deploying to production/github-mysqlutil June 11, 2017 05:26 Failure
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 11, 2017 05:28 Active
@shlomi-noach shlomi-noach deployed to production/github-mysqlutil June 11, 2017 06:09 Active
@shlomi-noach shlomi-noach had a problem deploying to production/hubot-slack June 11, 2017 06:09 Failure
@shlomi-noach shlomi-noach deployed to production/hubot-slack June 11, 2017 06:09 Active
@shlomi-noach shlomi-noach merged commit d1ec06f into master Jun 11, 2017
@shlomi-noach shlomi-noach deleted the force-master-recovery branch June 11, 2017 06:22
@jonahberquist
Copy link
Collaborator

Even better 👍

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants