fluent-bit Loki output plugin overrides fluent-bit escaped JSON parsing

[nottug]

Describe the bug
When using JSON logs with Docker, the JSON will be come escaped and stored in a JSON object like {"json": "ESCAPED_JSON_STRING"}. Native fluent-bit seems to handle this (and even has support with decoders for this exact issue), though when using the loki fluent-bit output plugin, the JSON remains as an escaped string (even when using decoders).

There seems to be solutions with promtail and Kubernetes for this issue, though I haven't been able to find a solution when using ECS with firelens. I ended up forking Loki and creating a new config option ParseJsonKeys which works well. I'm happy to open a PR here if that's an acceptable solution, this is the relevant commit.

To Reproduce
Steps to reproduce the behavior:

1. Install fluent-bit v1.8.9
2. Checkout loki at tag v2.4.1
3. Run make fluent-bit-plugin in the loki repo
4. Create two fluent-bit config files

# fluent-bit-native.conf

[INPUT]
    name dummy
    dummy {"log":"{\"level\":\"error\"}","stream":"stdout"}
    samples 1

[Output]
    Name stdout
    Match *

# fluent-bit-loki.conf

[INPUT]
    name dummy
    dummy {"log":"{\"level\":\"error\"}","stream":"stdout"}
    samples 1

[Output]
    Name grafana-loki
    Match *
    Url ${LOKI_URL}
    LineFormat json
    LogLevel debug

5. Run fluent-bit -e /PATH/TO/LOKI/clients/cmd/fluent-bit/out_grafana_loki.so -c fluent-bit-native.conf

[0] dummy.0: [1637114099.496755000, {"log"=>"{"level":"error"}", "stream"=>"stdout"}]

6. Run fluent-bit -e /PATH/TO/LOKI/clients/cmd/fluent-bit/out_grafana_loki.so -c fluent-bit-loki.conf

level=debug caller=loki.go:53 id=0 msg="processing records" records="map[log:{\"level\":\"error\"} stream:stdout]"

Expected behavior
For fluent-bit to unescape the JSON string and store it as native JSON instead of a string.

Environment:

• Infrastructure: ECS with firelens, fluent-bit v1.8.9, loki and loki fluent-bit output plugin v2.4.1
• Deployment tool: Terraform

[stale]

Hi! This issue has been automatically marked as stale because it has not had any
activity in the past 30 days.

We use a stalebot among other tools to help manage the state of issues in this project.
A stalebot can be very useful in closing issues in a number of cases; the most common
is closing issues or PRs where the original reporter has not responded.

Stalebots are also emotionless and cruel and can close issues which are still very relevant.

If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry.

We regularly sort for closed issues which have a stale label sorted by thumbs up.

We may also:

• Mark issues as revivable if we think it's a valid issue but isn't something we are likely
  to prioritize in the future (the issue will still remain closed).
• Add a keepalive label to silence the stalebot if the issue is very common/popular/important.

We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task,
our sincere apologies if you find yourself at the mercy of the stalebot.

[nottug]

This is definitely still an issue and there is a relatively simple fix unless I'm misunderstanding how fluent bit is meant to be used here.

[bbangert]

I'm running into this as well, it seems pretty common that people use JSON structured logs, its bizarre that the Loki output won't ship them right.

[duboisph]

Also running into this issue, however when disabling Merge_Log on the kubernetes filter we use, json is properly shipped to Loki.

[cyriltovena]

Made a PR very similar to your branch but wanted to avoid back and forth.

FYI Next time @nottug don't hesitate to send the PR we'll review it gladly. We love cool contributions.

[nottug]

That's perfect, thank you - and noted, I definitely will in the future. Thanks for getting this resolved, feel free to close this once its merged.