Promtail: Support CRI-O multi-lines

[eduan-envel]

Is your feature request related to a problem? Please describe.
Promtail has top-level support for CRI-O logs using the cri stage. However, this does not support multi-line CRI-O logs.

Describe the solution you'd like
The cri stage just handles multi-lines.

Describe alternatives you've considered
A custom multiline step. This is possible but the cri step removes some of the log tags so it doesn't work in conjunction.

Additional context
CRI-O has two log tags, P indicating a partial logline (one that will be continued by another partial logline, or by a 'full log line'), and F indicating a full logline.
Also, Fluent Bit already supports this. The documentation is here with some additional context in the feature request.

[kavirajk]

While working on PR to support cri-o multiline tag support #6177 in Loki, I found that cri-0 is not handling stacktrace or panics as partial log lines correctly.

Created a bug report here.cri-o/cri-o#5903

[kavirajk]

I think may be I misunderstood the purpose of log tags in cri-0?. may be cri-o split logs into multi lines only if it exceeds the log line threshold size and not during panic or stack-strace in the application.

[kekscode]

Yes correct. After 8192 bytes per line. Check also this issue cri-o/cri-o#4432

[dbluxo]

Is also described a bit here: cri-o/cri-o#4432 Nevertheless it would be very helpful to fix this issue, because a single line stack trace, which is over 8192 bytes long, is split by cri-o and ends up in Loki with several log lines. If the application already creates a multi line stack strace, I think you can solve it this way:
#74 / #3024

We even sometimes have the case that multiple log lines are subjected to the 8192 limit and are split: cri-o/cri-o#4432 (comment), here it would also help a lot if Promtail could join the fragmented messages.

[dbluxo]

It turns out that the split can theoretically happen at any position of a log line: cri-o/cri-o#4432 (comment)