Create and document rules of thumb for log levels in agones code

[roberthbailey]

As per #1220 we should come up with some general thoughts on what should be info and what should be debug for log lines in the agones code.

As mentioned during the last community meeting, too much logging can become expensive.

[aLekSer]

Some statistics of the log messages (total 300 rows were recorded), messages with count > 5:

Message	Count	Per cent
Processing	48	16.00%
Enqueuing	40	13.33%
Enqueuing immediately	38	12.67%
syncGameServerSet	27	9.00%
Reconciling GameServerSet	27	9.00%
syncGameServerSet finished	27	9.00%
Synchronising	22	7.33%
Update Fleet Status	20	6.67%
Added webhook handler	9	3.00%
Starting workers...	8	2.67%
Wait for cache sync	6	2.00%

Total 90.67% in above table out of 300 messages recorded.

Time is 07:48:24 - 08:01:51. Duration is 13 minutes. 22 log messages / minute.

Stats depends on the scenario of using Agones, but if we move some of the above messages into Debug we would reduce the amount up to a factor of 10.

https://docs.google.com/spreadsheets/d/1UYy6XELta2AU6X1P223xIlGsr2-aJXTpJEIdAZVVH4o/edit?usp=sharing

[aLekSer]

More real statistics from @KamiMay
#1218 (comment)

[markmandel]

I really like the https://reflectoring.io/logging-levels/ article that @pooneh-m suggested in the other thread! I'm quite tempted to suggest linking to that in the development guide and practically call it a day.

One rule I think should be add though - full dumps of an object (such as a GameServer), or JSON packet into logrus should only be done on Debug or Trace levels.

We do this a bunch! (or I did this a bunch I should say 😊 ).

[markmandel]

Maybe this is a tad heavy handed, but he vague rule of thumb I've been working on for the Agones controller at least is - if it has to do with the controller itself - starting, stopping, config etc - that's an Info level event.

If it has to do with a transient resource (fleet, gameserver etc) then that ends up being a Debug level event -- with the exception being if something goes wrong somehow (so errors are reported)

[aLekSer]

@akremsa, can you please look is there some places which should be logged with level down or up? And probably make a change to contributing.md to document all of those.

[akremsa]

Ok, I will take a look.

[markmandel]

@akremsa the sdkserver should be a great place to start. I feel like there is a large chunk of Info logging that could be moved to debug 👍

[markmandel]

This looks like it's completed! If nobody objects, I'll close this next week.