Audit logging messages and map them to Debug/Info/Error

[scrye]

It would be useful to remove the tracing logging level, because:

• the formatting of trace log lines is not consistent with the formatting of other levels;
• it requires complex logic/hacks in the go/lib/log package, because the underlying logging library does not support custom levels
• it provides no tangible benefits; five logging levels should be enough, a sixth does not add much

As a first step, we should reevaluate all logging calls in the code base, and decide what level they should be. As a general guideline, the following could be used:

• Debug: very low-level details about what the application is doing; target audience is developers familiar with the code base; for everyone else, it would be challenging to understand them. Everything Trace log entry in current code should be bumped up to Debug.
• Info: high-level details about what the application is doing; target audience is developers and operators with knowledge of SCION and application APIs. Some Debug log entries in current code should be bumped up to Info.
• Warn: details about events that do not impact the application's functionality, but are unexpected during normal operation
• Error: details about events that might impact the application's functionality, but from which the application was able to recover
• Critical: details about events that severely impact the application's functionality, including events that cause it to crash

[scrye]

@worxli, @mkowalski, @karampok: Does the proposal sound reasonable to you?

[scrye]

Switching this from a proposal to a planned change.

[oncilla]

I would like to have a guideline written in our docs that covers the following:

• When to use which log level (I think the description in this issue covers this pretty well)
• Interaction between logs and metrics: More precisely: When to use logs vs when to use metrics

[scrye]

I would like to have a guideline written in our docs that covers the following:

* When to use which log level (I think the description in this issue covers this pretty well)

* Interaction between logs and metrics: More precisely: When to use logs vs when to use metrics

Extracted to Anapaya/scion#3072 (sorry, internal link).

[scrye]

Let's go even further with the reduction in logging levels. As discussed in Slack, we can use three levels: Debug, Info, Error.

From Slack (sorry, internal link):

• Debug: developer only, very low-level details
• Info: what the application is doing, quite verbose, includes “errors” that are part of normal operation. A lot of Errors from today would move here, because they’re confusing to third-parties who might be genuinely concerned by some of these, even though they’re part of normal operation.
• Error: application encountered a major problem (e.g., DB unreachable), and it might even need to terminate.

[scrye]

To have this actionable:

• All Trace logging calls become Debug
• All Debug calls are re-evaluated, and some become Info
• All Info calls stay Info
• All Warning calls become Info
• All Error calls are re-evaluated, and some become Info
• All Crit calls become Error

Library go/lib/log is also refactored to only expose Debug, Info, and Error APIs. Trace is removed from internal logic.

[scrye]

None are of the above recommendations are set in stone, so we'll need to take a look at each of them and decide what makes most sense, maybe even editing the message and arguments.

[scrye]

Let's tackle this for go/lib first, and see how much work that is. If it's too much, we'll break it down further.