Lifecycle tweaks for Singularity

[ssalinas]

Small one:

• Make sure to cancel in progress task reconciliation on shutdown

Bigger one:
Currently when a new leader takes over we can hit an order of events like the following:

• Gains leadership
• Other api instances see this and start sending writes to new leader
• leader cache bootstrapping starts
• leader cache bootstrapping loads type X (e.g. requests, deploys, etc) but is not yet activated
• Writes make it through to zk, but not to leader cache because active() is false
• leader cache finishes bootstrapping

We now have a case of a write for type X that made it to ZK, but not to the leader cache. This can result in missing deploy data, missing requests, requests that are present but actually were deleted, etc.

The new PR here adds an additional check within active() that should serve to block the calling thread when bootstrapping of the leader cache is still in progress. So, we will delay all incoming writes and wait for this to finish, rather than trying to reconcile state afterwards.

Would like feedback on the java-y bits here for sure:

• Is this performant enough during regular operation? In staring it down it should just be one additional if statement on a volatile boolean
• Any race conditions where the calling thread could get stuck permanently? I think I covered this with the while loop and synchronized objects, but want to be sure

[rosalind210]

what would this do?

[ssalinas]

bit of background on wait + notify in https://www.baeldung.com/java-wait-notify. Though realizing I probably need notifyAll here actually. Leaving as a TODO for later

[rosalind210]

🚢