Feature: multi-line logs #74
Comments
|
Agreed, thanks @ventris. Any idea how to do this though? I guess we could have the option to parse the log entries in promtail, and define each entry as starting with a time stamp? |
|
Some applications require an "special" character before the multi-line event. I know that our current Wildfly setup puts #### before the multiline event. But I think that parsing the log entries somehow and cut each entry with the timestamp as key is the better solution. |
|
Splunk handles this in a smart way. Source: extract-timestamps in splunk Maybe somethink like that? One thing to take into account with only look for timestamps is that they can occur within a stacktrace. That is why Splunk has a variable that you specify how many characters into a line it should look for the timestamp. |
|
Here is another multi-line scenario Our log files are JSONL (http://jsonlines.org/) and look like this: We'd like to (1) associate the key-value metadata with each log line, (2) ingest the array of log lines from the "messages" key and (3) ideally group the associated log lines (so that a log reader client could easily show them together). If it makes ingestion easier, we could easily keep a timestamp per message in the array instead. Something like this: Also, Loki sounds awesome! 🏆 Have been looking for a lightweight alternative to the ELK stack for logging. We could easily live without full-text search inside log messages (distributed grep would be enough). However, we'd need filtering by keywords and multi-line log handling to make it work. |
|
We recently merged support for having multiple entries with the same timestamp; when running the docker log parser, multi-line logs usually show up as entries with the same timestamp, and are therefore grouped together. Not a perfect solution, but helps. |
|
When I was at Logentries we delimited on |
|
My understanding is that loki doesn't treat the content of log "lines" specially, so it should be up to the process tailing the logfile to decide when a single loki log "line" should actually contain multiple newline-delimited "lines" of content (maybe it would have made more sense to call it a log entry but such is life). Of course I may be misunderstanding the intent here, but it seems like something like a backtrace would be more useful as a single entry in loki rather than being split up. |
|
From what I've seen in other similar log tailing (fluentd concat plugin), the tailer provides options for multi-line start regexp (lines matching the regex indicate the start of a new log entry, so flush the old one out), with a timeout/max buffer size (if x lines end up in the buffer without a new log entry appearing, flush the logs out anyways). I implemented support for the multi-line start regex for a hackathon, and it seems to work decently. I'll try to clean up the code over the weekend and open a PR to show as an example. I didn't implement any sort of timeout/max buffer size though. |
|
Opened a PR for this issue |
|
Note that multi-line is different depending upon the EntryParser. With CRI, it will send multiple entries with different flags to indicate if it's a partial log or a full log. See this document. |
|
Currently investigating this. |
|
I looked at it #399. There would be a lot of refactoring involved if we want to get this feature in the current pipeline, I would love to hear more about how this is useful before getting started. Basically how merging lines together helps is it a visual problem? Or is to improve storage of those ? Good example of workload emitting multiline logs ? I want to know a bit more. |
|
Java and C# stacktraces are multiline. One of the problems with not having access to the full event is that it makes it much harder to search for other similar events because that the linebreaks can cut javaclasses in half and thus does not paint a real picture of what happened in the logs on the server. Another thing is if the example below is handled with multiple logs instead it is really hard to search for say:
If I search for them withing having the stacktrace in one entry I will just get the rows that matched my keywords and not the whole stacktrace. This makes troubleshooting java much harder with Loki if multiline events are handled as-is. Because I first need to search for the keywords and then try to find the first and last entry of the stacktrace so I can read it as a whole. And the stacktraces can be several hundred lines. I think there are other multiline events that should benefit from having it all in the same entry, but I have only worked with Java that behaves like that so I can not talk for other languages. Here is an example stacktrace from java: |
|
Above, plus Grafana only shows them in descending order by time, so stacktrace is upside down from how it should appear. Also makes emoji art impossible to see, which a lot of java apps print on startup. :) |
|
Python stack traces are the same: It's all about the context. Only complete log record holds the whole context, not any single line. |
|
Could you please encounter the following scenario or at least give your opinion: Because of Loki we have changed the log format of our appenders to use the logstash JSON-layout-formatter in our logback configuration. That's why now we have an all information containing log entry with also the whole stacktrace (containing the already noted \n\t). So no need to join lines afterwards. That sounds good to me... Here is a sample entry: But... In loki and the file log itself the stacktrace isn't readable anymore. That's not good. At least for the loki side maybe something like a configuration possiblity (fluentd plugin plus promtail) for fields would be good (e.g. multiline_fields: [stack_trace]). Then in grafana each multiline field could be represented in a well formated column maybe (just a simple thought). I don't know if the one-json-line approach is the best one but just wanted to note it here. In theory it might also be well displayed. For us this would be the most needed Loki+Grafana feature since I wouldn't let other engineers work with a one-line stacktrace.. On the other side we also need to handle logs from older environments where multilines in spite of stacktraces would play a role: PS: Also this issue #616 also seems to be related with multilines. |
|
We checked with Azure Log Analytics and with Kibana.. If a json formatted log entry stacktrace arrives in
|
|
This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions. |
cyriltovena
added
the
keepalive
stale
bot
removed
the
stale
|
Using If we use example from #74 (comment) Then Now tracked in #1093 |
|
Unfortunately as of today this is still an issue with grafana/loki:2.1.0 |
|
Have you tried to configured promtail to match those multi lines ? https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/ |
Further I have documented below, the way I have setup loki |
|
It has not been release into 2.1, it will be in 2.2. You can use |
|
grafana/promtail:master-ee9c629 doesn't work fine for me. I have crashloopbackoff and this in pod logs goroutine 1115 [sleep]: goroutine 1116 [chan receive]: goroutine 1117 [select]: goroutine 1118 [select]: goroutine 1119 [syscall]: |
|
The log message is cut off could you share the full error log ? Have you tried recent build I fixed a panic recently in promtail that was unreleased. |
@cyriltovena can you tell me when 2.2 is gonna be released? |
|
Hi all. I'm also facing similar issues as @mnadeem does. I was following the docs (https://grafana.com/docs/loki/latest/clients/promtail/stages/multiline/) and added this configuration to my promtail values file (I'm installing the promtail helm chart in my cluster): But promtail doesn't start: Using promtail v2.1.0 and helm chart version 3.1.0: So, this will be fixed in v2.2.0? We need to wait for now? |
|
@antoniocascais - I think so, it's lined up to be released in
|
|
We wanted to release 2.2 this week. I don't know if we're on track but if not next week. We wanted to include some read path bug in there too. |
|
Does this capability work when scraping the system journal or is it only when scraping files? I have an application dumping tracebacks line by line into the system journal, was hoping to pull those together into one log line in Loki. I've been trying to use it but seems like its getting ignored. Or do you have to combine it with |
|
Yes it does |
|
Ah I figured it out. There was an escape sequence at the front of the text for terminal color-coding and that was causing my regex to fail. This is great! Much nicer to look at 👍 |
|
Just incase anyone else is watching this space for the update, this was all included in v2.2.0 which is now available and released: See here: https://github.com/grafana/loki/releases/tag/v2.2.0 |
|
If someone here is still alive, maybe you can help me. Is there an option to collapse the error stack trace, and if I need to see it I just click on the arrow sign and see a well-formatted stack trace? |
|
anyone had success multilining with logback and spring boot ? |
|
👋 I wrote the multiline stage. @omelnyk10 and @Jojoooo1, you both want to collapse Java stack traces? |
|
@jeschkies yes |
|
Exactly @jeschkies ! With the logback default encoder! |
|
There is an example in the documentation for configuring logback. Does that help? |
|
Made it work with |
Update from upstream repository
|
Anyone can share a sample I have tried lots of options without success. This is my last piece of config: TBH, the grafana docs, in general are hard to follow. Docs are modular in the sense that they are scattered according to the respective config sections. You can get sample blocks but not a full sample of the agent configuration file. Therefore, it is sometimes hard to see where you are making a mistake. |
I tried the following in my values.yaml: snippets:
pipelineStages:
- cri: {}
- multiline:
firstline: '^\d{4}-\d{2}-\d{2}'
max_wait_time: 3sWith helm chart: - name: promtail
version: 6.14.1
repo: https://grafana.github.io/helm-chartsBut unfortunately it is not working at all. Still all logs are collected line by line no summarisation on timestamp... Any ideas does it still work for you? @Jojoooo1 |
and others
Currently when an multi-line event is written to a logfile promtail will take this as each row is its own entry and send the separately to loki.
It would be great if Loki could handle multi-line events due to stacktraces.
This might also be a bug if promtail already should handle multi-lines.
The text was updated successfully, but these errors were encountered: