Controllers with inconsistent state can cause intermittent failures

[stuartwdouglas]

This is a follow on issue for the PR #2790 that adds a delay before deleting runners.

Fundamentally the issue is about deciding when a deployment is 'ready', and when old deployments can be removed. There are other cases that we need to consider here as well, e.g. starting async calls before all controllers have their route tables updates can result in failure.

We need to remove the sleep based hack and come up with a proper solution around deployment readiness.