Add a notion of 'time' to some server outputs

[KtorZ]

Why

Because the whole communication API for Hydra is asynchronous, it can be hard for a client application to know how to relate events to time. For example, GetUTxOResponse returns the current UTxO known of the head; but "current" is not a well-defined moment. What was the last confirmed snapshot number at that moment? What was the last seen transaction? What was the virtual slot of the head? Etc..

What

Most server outputs from the API would benefit from some kind of anchor that allows to correlate them better with the state of the system. Once we start working on #196 we may have introduced a notion of slot / time inside the head which could come in handy. In the meantime, it may be interesting to link information to the latest snapshot so that replies from the server can be better associated with (verifiable) moments of the system.

[ch1bo]

What is it what we should do concretely? Add slotNo to all server outputs?

When trying to debug #388 it also occurred to me that we should include the current slot in chain related logs - might be related.

[KtorZ]

A slotNo or a snapshot number. Maybe both. I think both have values.

[ch1bo]

This is also relevant to debugging some issues I have right now where the fanout validator decides "it's not time yet", but it's not clear from the logs our server outputs whether that's true or not.

[ch1bo]

Was also part of feedback by @Quantumplation. Especially in SnapshotConfirmed messages a notion of time would be good to have.

[Quantumplation]

Another nuance we teased out is that outwardly (hydra node to rest of the world) we really want a fairly high granularity real world time, but inwardly (in the script context) we can tolerate different notions of time.

[ch1bo]

This discussion (and soon issue on our roadmap) is about the outward direction as far as I can tell. I would propose to just use node-local getCurrentTime :: IO UTCTime, which will provide picosecond precision wall-clock measurement. It's possible that you will see SnapshotConfirmed messages with different timestamps between different nodes.

[ch1bo]

We have added sequence numbers and timestamp fields to all server outputs with #618. @KtorZ @Quantumplation do you think this addresses this conversation in satisfied way? Note that we have other items for capturing related ideas/concerns #196 and #416.