-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug] Newlines in logfmt sometimes work, sometimes don't #3927
Comments
Thanks for the report. If you can send a PR I'll take a look, I'd like to make sure this has no side effects. |
I'm wondering if
is a valid quoted string ? I would have expected
|
Why isn't it valid though. Loki uses json to encapsulate a log line. It is impossible for it to -not- be valid as the delimiters have already been found. However if you would prefer to think of this as an X Y problem, the bigger issue is actually that loki doesn't take non-label attributes, so we are forced to add them as part of the log line and I used logfmt instead of double-encoding json, which is more readable at a glance. It would be trivial for loki to work with non-indexed labels by appending them to the log line and transparently parsing them on load, but that's a much bigger discussion and for now I'd rather just ship some logs without needing to write a bunch of encoders thanks. |
Not sure what you mean about json here ? I'm not against the change I'm trying to fully understand it. |
What I mean is loki stores logs by 'message', not by 'line'. When you send logs to loki it does not care if there are newlines in the log, because it is already encoded in json when sent. The original log aggregator using logfmt was some heroku tool that parsed individual lines only, so it needed escapes. A token containing |
If you have a literal "control char" in your log, encased in logfmt quotes like so:
It works fine when put through logfmt. And I can't really see a problem with that.
However the parser does see a problem, but this logic only applies if it first detects an escape sequence:
loki/pkg/logql/log/logfmt/jsonstring.go
Line 122 in a9d85de
So this fails to parse because of the newline:
Unfortunately the only control char I escape in my logs is a literal backslash or quote. I didn't see a need for any other form of escapes because each line is already encapsulated in json when I send it to loki.
Simply removing the check on
c < ' '
fixes the issue, still passes all tests, allows literal newlines, looks better in loki when scrolling through unformatted lines, has consistent behavior, uhhh and saves a few picoseconds of CPU.Here's some tests you could put in parser_test.go:
The text was updated successfully, but these errors were encountered: