[Spot] sky spot cancel does not interrupt PENDING or STARTING job

[Michaelvll]

The current behavior of our code is that we will set the spot job status for PENDING jobs to STARTING first before setting it to CANCLLED, which can be a bit confusing.

[Michaelvll]

Our sky spot cancel does not interrupt the provisioning and setup of a spot cluster either:

It seems that the spot cancel command isn’t propagated immediately.

(base) MacBook-Pro-3:~ dementrock$ sky spot cancel 28
Cancelling managed spot jobs with IDs 28. Proceed? [Y/n]: y
I 11-14 05:44:43 spot_utils.py:149] Cancelling jobs 28.
Job with ID 28 is scheduled to be cancelled within 20 seconds.
but the job is still in STARTING state for a while (still in setup phase).

[romilbhardwaj]

Reopening with #1432