Add system logs to increase visibility

[surajkota]

As a user, I want to get all system logs corresponding to a benchmark in addition to the benchmark script logs because it helps root cause the problem faster

Related #996

[gavinmbell]

Can we better define "system logs"? What kind of information is this? Who emits them (are they low level logs like what you would find in /var/log/messages kind of thing)? And/Or the output from the Anubis services themselves from the processes that happen between event ingress and egress?

One of the first things to look at is what is the structure of the information we are putting into logs. (it should be structured - pretty much just like an event) . This begs the question of how do we determine the heuristic we want to appy to discern what should be logs and what should be events. I would assert that logs have no semantic value - there is nothing you can do with them. They provide context.

let's sort this then we can start emitting accordingly.

[surajkota]

terms used: client-id+action-id = request-id

Anubis consists of multiple services and service instances that are running on multiple machines and requests often span multiple service instances. Most services operate on publish/subcribe mechanism and perform one or more operations to handle a request.

We have events passing through the system such that request-id is passed to all services that are involved in handling the request. (If not we need to add it)

We need to instrument services with code that includes the request-id in all log messages, operations performed when handling a request in a the service and maybe records information (e.g. start time, end time)

It enables developers to see how an individual request is handled by searching across aggregated logs for each request id

What kind of information is this? - Each service instance generates writes information about what it is doing. The logs contain errors, warnings, information and debug messages etc

Who emits them (are they low level logs like what you would find in /var/log/messages kind of thing)? And/Or the output from the Anubis services themselves from the processes that happen between event ingress and egress? The later