Fluent-bit plugin encoded characters

[nomatterz]

Describe the bug
When sending events (which are Java Logs in json) from Fluent-bit to Loki encoded characters within record are shown as plain text in Grafana. For example instead of newline there is \n
I think there is some issue with plugin itself as fluent-bit logs are ok.
Also tested the same log messages with FluentD and FluentD Loki plugin - everything is ok in Grafana

To Reproduce
Send JSON Log Message with stacktrace (Java exceptions for example)

Expected behavior
Grafana displays newlines and tabs instead of \n \t

Environment:
Fluent-bit-->Loki<--Grafana

• Infrastructure: Kubernetes
• Deployment tool: helm (loki/fluent-bit, loki/loki charts)

Screenshots, Promtail config, or terminal output
Fluent-bit logs:

[1] kube.var.log.containers.cst-gap-6f99986fff-hpzbt_cst-gap-dev_cst-gap-dd17add6b3620f756565aedfc9f3aae578fdf09f22e641c6e0fc3d458c5b94c9.log: [1592739931.069359238, {"app"=>"spring.application.name_IS_UNDEFINED", "ts"=>"2020-06-21T11:45:31.031Z", "logger"=>"org.zalando.problem.spring.web.advice.AdviceTrait", "level"=>"ERROR", "class"=>"org.zalando.problem.spring.web.advice.AdviceTrait", "method"=>"log", "file"=>"AdviceTrait.java", "line"=>246, "thread"=>"http-nio-8088-exec-3", "stack"=>"com.intellecteu.miml.ug.annotationsui.controller.errors.InternalServerErrorException: Could not fetch collections from database
	at com.intellecteu.miml.ug.annotationsui.utils.IOUtils.convertStreamIntoCollection(IOUtils.java:91)
	at com.intellecteu.miml.ug.annotationsui.service.GridFsServiceImpl.lambda$fetchCollections$1(GridFsServiceImpl.java:100)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.Iterator.forEachRemaining(Iterator.java:116)
	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
	at com.intellecteu.miml.ug.annotationsui.service.CollectionServiceImpl.getCollections(CollectionServiceImpl.java:167)
...

Fluent-bit config:

 [SERVICE]
        HTTP_Server    On
        HTTP_Listen    0.0.0.0
        HTTP_PORT      2020
        Flush          1
        Daemon         Off
        Log_Level      warn
        Parsers_File   parsers.conf
    [INPUT]
        Name           tail
        Tag            kube.*
        Path           /var/log/containers/*.log
        Parser         docker
        DB             /run/fluent-bit/flb_kube.db
        Mem_Buf_Limit  5MB
    [FILTER]
        Name           kubernetes
        Match          kube.*
        Kube_URL       https://kubernetes.default.svc:443
        Merge_Log On
        K8S-Logging.Parser Off
    [Output]
        Name loki
        Match *
        Url http://RELEASE-NAME-loki:3100/api/prom/push
        TenantID ""
        BatchWait 1
        BatchSize 10240
        Labels {job="fluent-bit"}
        RemoveKeys kubernetes,stream
        AutoKubernetesLabels false
        LabelMapPath /fluent-bit/etc/labelmap.json
        LineFormat json
        LogLevel warn
  parsers.conf: |-
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

  labelmap.json: |-
    {
      "kubernetes": {
        "container_name": "container",
        "host": "node",
        "labels": {
          "app": "app",
          "release": "release"
        },
        "namespace_name": "namespace",
        "pod_name": "instance"
      },
      "stream": "stream"
    }

Grafana output with Fluent-bit as collector and shipper

Grafana output with FluentD as collector and shipper

[cyriltovena]

It’s different for Promtail, open an issue with more details/examples please.