Improvements in disaster recovery

[selzoc]

Hi BOSH friends!

I wanted to start a discussion about a feature we've been working on that promises to speed recovery in IaaS disaster scenarios. We've been calling this feature bosh recover.

bosh cloud-check is great, but it can be frustratingly slow for situations where there are a large amount of VMs down or unhealthy due to underlying IaaS failures. In most cases we've seen, operators have opted to use an awkward combination of bosh ignore, bosh recreate, jq and bosh deploy to try and get their VMs back online faster. In those scenarios, the interactions between all of the commands and also the state in the director is very hard to reason about. This is especially the case if you also have a failed deployment.

Much of the underlying functionality of bosh cloud-check is sound and will be re-used for this feature. In particular, the GET and PUT /deployments/:deployment/problems endpoints are featured.

The new feature consists of:

• Improvements to the GET /deployments/:deployment/problems endpoint to return the instance_group of each problem
• Improvements to the PUT /deployments/:deployment/problems endpoint to allow overriding max_in_flight per instance group
• New bosh create-recovery-plan command, which scans a deployment for problems, prompts the user to select resolutions per problem type per instance group, optionally override max_in_flight for the instance group, and then writes out a YAML file with those resolutions.
• New bosh recover command, which takes the plan generated by bosh create-recovery-plan and applies it to the given deployment.
• Accompanying bosh.io docs

Note that the recovery plan has no deployment-specific information in it. This allows you to use the same plan for multiple deployments. An example of this we've seen is for on-demand service-broker deployments, where the deployment name is generated, but the instance_groups inside of the deployment are all the same.

[cunnie]

w.r.t. @beyhan 's comment, "how this relate to the existing cloud-check command? Should we deprecate it in favour of this one at some point?"

I don't think we should deprecate cloud-check because for most of the cases cloud-check is the easier path, rather than running the two-step bosh create-recovery-plan and bosh recover.