move to Instant.elapsed() instead of time::Sleep timer

[bmc-msft]

In current tokio, time::sleep().elapsed does not update unless the Sleep
is polled. as such, the execute_pending_commands never fires. This
replaces the sleep().elapsed with Instant.elapsed().

[ranweiler]

Maybe we should update this to select! in a loop, against two Interval futures that explicitly control how often we attempt to execute pending commands or update the state machine. Something like:

let cmd_interval = time::interval(PENDING_COMMANDS_INTERVAL);
let update_interval = time::interval(UPDATE_INTERVAL);

loop {
    self.heartbeat.alive();

    tokio::select! {
        _ = cmd_interval.tick() => {
            self.execute_pending_commands().await?;            
        }
        _ = update_interval.tick() => {
            if self.update().await? {
                debug!("agent done, exiting loop");
                break;                                
            }
        }
    }
}

[ranweiler]

We'd probably want to update the busy() state impl too, so it checks if an Interval has ticked before polling the remote queue.

[ranweiler]

Also, we'd want to replace this sleep() with an Interval. The logic we'd want for the ready state is: "every UPDATE_INTERVAL, give the inner state handler an opportunity to check for available work. The state handler will then consult its own timer, so there is at least WORK_POLL_INTERVAL seconds between calls to the remote work queue."

[bmc-msft]

It's underlying issue is made complex based on how the state machine is implemented.

• Free needs to sleep because it's response is dependent on the server.
• Busy shouldn't sleep, but instead await on the output of the workers.
• All of the other states should proceed immediately without delays.

In my opinion, the state machine's delay logic belongs in each of the individual states and instead of a loop where we manage the transitions, I think we should move to a model akin to the A Fist Full of States from Deis Labs.

[bmc-msft]

Either way, that's a large amount of work and should be a follow-on PR.