Don't block worker while evaluating a policy

[lgfa29]

The Nomad Autoscaler v0.0.x had a single goroutine evaluating policies. When we introduced multiple checks in the v0.1.0 release we tried to parallelize as much work as possible to keep the goroutine as fast as possible.

This created a race condition where the worker and the checks could be stuck waiting on each other.

In the v0.2.0 release we introduced the EvalBroker and Workers, so parallelization can now be provided at a higher level (per policy).

This PR turns the policy evaluation process into a linear execution to prevent the Worker from getting stuck. It also introduces checks for ctx.Done() before potentially long-running operations happen (namely, around plugin calls).

Future work will include a heartbeat mechanism to detect and recreate workers that get stuck, then we can start parallelizing the check executions again.

Closes #218 #303 #343

[cgbaker]

nice. there are a number of places where we perform unchecked type assertion after calling dispense. i'm willing to be convinced otherwise, but it seems like a good idea to do a checked type-assertion. a bug somewhere earlier in the code shouldn't be able to result in a panic here.

i only marked the first of these.

[lgfa29]

nice. there are a number of places where we perform unchecked type assertion after calling dispense. i'm willing to be convinced otherwise, but it seems like a good idea to do a checked type-assertion. a bug somewhere earlier in the code shouldn't be able to result in a panic here.

i only marked the first of these.

The PluginManager should be checking those, but I agree, it's better to protect against panics here.

I think we could expand the PluginManager to request specific plugin types now that we are mostly settled on the types of plugin we have.