Snap spawning multiple plugin processes

[nanliu]

When debugging #1448, we tried to suspend/resume process via kill -STOP & kill -CONT. This uncovered some issues with how we start plugin processes. After several heartbeat failures, if the plugin process is resumed, snap will terminate it, but spawn multiple plugin processes.

We should guard this by ensuring this block of code is not executed simultaneously by multiple plugin failure events:
https://github.com/intelsdi-x/snap/blob/master/control/runner.go#L246-L281

This can be done crudely by a mutex lock and defer unlock around that section of code.

[lynxbat]

I would recommend locks be used for start, restart, and stop. I would have read and write locks on this also. This will prevent issues with comparisons like https://github.com/intelsdi-x/snap/blob/master/control/runner.go#L247 being wrong while a write is in flight.

[lynxbat]

We should capture a repeatable test from #1448 discovery to replicate and prove this is fixed. I am sure we were. Just pointing out the obvious :)