Skip to content
This repository has been archived by the owner on May 13, 2024. It is now read-only.

Add system logs to increase visibility #1001

Open
surajkota opened this issue Jan 7, 2020 · 2 comments
Open

Add system logs to increase visibility #1001

surajkota opened this issue Jan 7, 2020 · 2 comments
Labels
enhancement New feature or request

Comments

@surajkota
Copy link
Contributor

As a user, I want to get all system logs corresponding to a benchmark in addition to the benchmark script logs because it helps root cause the problem faster

Related #996

@surajkota surajkota added the enhancement New feature or request label Jan 7, 2020
@gavinmbell
Copy link
Contributor

gavinmbell commented Jan 8, 2020

Can we better define "system logs"? What kind of information is this? Who emits them (are they low level logs like what you would find in /var/log/messages kind of thing)? And/Or the output from the Anubis services themselves from the processes that happen between event ingress and egress?

One of the first things to look at is what is the structure of the information we are putting into logs. (it should be structured - pretty much just like an event) . This begs the question of how do we determine the heuristic we want to appy to discern what should be logs and what should be events. I would assert that logs have no semantic value - there is nothing you can do with them. They provide context.

let's sort this then we can start emitting accordingly.

@surajkota
Copy link
Contributor Author

terms used: client-id+action-id = request-id

Anubis consists of multiple services and service instances that are running on multiple machines and requests often span multiple service instances. Most services operate on publish/subcribe mechanism and perform one or more operations to handle a request.

We have events passing through the system such that request-id is passed to all services that are involved in handling the request. (If not we need to add it)

We need to instrument services with code that includes the request-id in all log messages, operations performed when handling a request in a the service and maybe records information (e.g. start time, end time)

It enables developers to see how an individual request is handled by searching across aggregated logs for each request id

What kind of information is this? - Each service instance generates writes information about what it is doing. The logs contain errors, warnings, information and debug messages etc

Who emits them (are they low level logs like what you would find in /var/log/messages kind of thing)? And/Or the output from the Anubis services themselves from the processes that happen between event ingress and egress? The later

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants