Expose GameServer state change metrics

[pjayara-g]

I would like to know how long GameServers spend in a given state - for example how long does a game session last i.e. how long does it take to go from Allocated to Shutdown or how long does it take for a GameServer to become Ready. This is very helpful during deployments where a dramatic change might be a cause for concern.

I discussed this with @markmandel and we have two possible approaches:

1. We could add code to watch events. We might be able to use ListOptions to filter it down to events about GameServers. Then the code maintains an internal cache of state -> timestamp for each game server. The cache might evict based on LRU. LRU would mostly work if we expect game servers to roughly run for the same time.
2. Modify the GameServer CRD by adding a list of <state, timestamp> pairs to it. State updates will append to this list.

[markmandel]

I expect @Kuqd might have some thoughts on this?

[pjayara-g]

@roberthbailey suggested watching game servers instead of events.

[markmandel]

Other thought - we are on v0.18.0 of OpenCensus. Not sure if a more up to date release would make this easier?

May be worth investigating?