Log aggregation issues because of new lines

[AlianBenabdallah]

Summary

Loki is a horizontally scalable, highly available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate. It does not index the contents of the logs, but rather a set of labels for each log stream.

Loki reports errors for incorrectly formatted logs. A quick fix would be to remove every "\n" from the logs.
We should also leverage log labelling. Example of a labeled log :

info!(
            label1 = "foo",
            label2 = "bar",
            "random message"
);

Description

Hermes' logs can be shipped to Loki by two different ways but, both methods will throw errors :

• Redirect Hermes' output to hermes.log and ship the log's content to Loki.

Standard logs can not be parsed by Loki's default config because of escaped newlines and/or tab sequences in log lines. Loki will throw an error Your logs might have incorrectly escaped content. It can not know which line belongs to which log.

Using a JSON output fixes some issues. Loki can link a line and its log. However, it still reports error because of escaped characters and some content is not correctly parsed.

• Using the crate tracing-loki. Do not try this crate with the example in the docs, instead use this example.
  
  The field message was not correctly formatted. Also, using tracing-loki is most likely not the right approach because it would bind users to Loki while other log aggregators exist.

How to use Loki ?

Install through Docker

https://grafana.com/docs/loki/latest/installation/docker/

wget https://raw.githubusercontent.com/grafana/loki/v2.6.1/production/docker-compose.yaml -O docker-compose.yaml
docker-compose -f docker-compose.yaml up

Grafana

Open http://localhost:3000
Login : admin, password : admin
Select Add your first Data Source
Select Loki
Enter URL : http://loki:3100.
Press Save & Test.

Explore

Click on Explore on the left of the screen.

(Note : do not use localhost:3100)

The content of every log file in '/var/log' is shipped to Loki by Promtail.
To the right of "Run query", select 5s to automatically run a query every 5s.

Send a logfile to Loki

To listen to osmosis-1, use the config file in #2533 .

sudo touch /var/log/hermes.log
sudo chown user /var/log/hermes.log
hermes listen --chain osmosis-1 &> /var/log/hermes.log
or 
hermes -- --json listen --chain osmosis-1 &> /var/log/hermes.log

Query your logs

Come back to Grafana and select filename = var/log/hermes.log.

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate milestone (priority) applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[adizere]

Great work with documenting this Ali! I made a note to ask operators whether Loki is of interest to anyone in particular, which will help prioritize this work here going forward.

In general, we should likely continue improving logs, eg #1538 #2544 !