Periodic healthcheck improvements

[ahreehong]

Issue #, if available:

Description of changes:

1. Fix bug where if machine passes healthcheck, is not deleted from queue to be deleted. Add delete(currClusterHFConfig.unhealthyMembersToRemove, endpoint) to line 150 in periodic_healthcheck.
2. Limit number of machines to be deleted in healthcheck to delete one machine at a time and always wait for the new replica to be created before deleting another one. This should be easy to do by just checking what's the desired number of replicas.
3. To make sure we preserve quorum, check if it's safe to remove a machine as a member: currentTotalMembers - unhealthyMembers >= currentTotalMembers/2 + 1. Here currentTotalMembers and unhealthyMembers needs to take into account all machines that part of the etcd cluster and not only the owned ones.
4. Perform healthchecks iterating over the all machines instead of status.endpoints. Endpoints in the status might not always be updated. In fact with the way the code is structured, the status is not updated until all the machines marked for deletion are successfully deleted. Ignore machines in bootstrapping phase.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[abhinavmpandey08]

/approve
/lgtm
/woof
/goaty-goat-goat