Missing fields and unparsed logs

[kamaraj-muthupandian]

We are forwarding our logs through fluent-bit log shipper to humio, post container runtime migration from docker to CRI we have noticed that few fields are missing and not able to see the logs in json format. Below the config, please help advising on this issue.

Missing fields:

log_processed.log
log_processed.stream
log_processed.time

Parser Config

custom.conf: ""
 fluent-bit-filter.conf: |-
   [FILTER]
       Name                kubernetes
       Match               kube.*
       Kube_Tag_Prefix     kube.var.log.containers.
       Kube_URL            https://kubernetes.default.svc:443
       Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
       Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
       Merge_Log           On
       Merge_Log_Key       log_processed
       K8S-Logging.Parser  On
       K8S-Logging.Exclude On
       Cache_Use_Docker_Id On

 fluent-bit-input.conf: |-
   [INPUT]
       Name             tail
       Path             /var/log/containers/*.log
       Parser           cri
       Tag              kube.*
       Refresh_Interval 5
       Mem_Buf_Limit    5MB
       Skip_Long_Lines  On

 fluent-bit-output.conf: |-
   [OUTPUT]
       Name  es
       Match *
       Host ${FLUENT_ELASTICSEARCH_HOST}
       Port ${FLUENT_ELASTICSEARCH_PORT}
       tls ${FLUENT_ELASTICSEARCH_TLS}
       tls.verify ${FLUENT_ELASTICSEARCH_TLS_VERIFY}
       HTTP_User ${HUMIO_REPO_NAME}
       HTTP_Passwd ${HUMIO_INGEST_TOKEN}
       Logstash_Format On
       Retry_Limit False
       Type  flb_type
       Time_Key @timestamp
       Replace_Dots On
       Logstash_Prefix FluentBitHelmChart
       Buffer_Size 5MB
       compress gzip

 fluent-bit-service.conf: |-
   [SERVICE]
       Flush        1
       Daemon       Off
       Log_Level    info
       Parsers_File parsers.conf

 fluent-bit.conf: |-
   @INCLUDE fluent-bit-service.conf
   @INCLUDE fluent-bit-input.conf
   @INCLUDE fluent-bit-filter.conf
   @INCLUDE fluent-bit-output.conf
   @INCLUDE custom.conf

 parsers.conf: |-

   [PARSER]
       Name   apache
       Format regex
       Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
       Time_Key time
       Time_Format %d/%b/%Y:%H:%M:%S %z

   [PARSER]
       Name   apache2
       Format regex
       Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")?$
       Time_Key time
       Time_Format %d/%b/%Y:%H:%M:%S %z

   [PARSER]
       Name   apache_error
       Format regex
       Regex  ^\[[^ ]* (?<time>[^\]]*)\] \[(?<level>[^\]]*)\](?: \[pid (?<pid>[^\]]*)\])?( \[client (?<client>[^\]]*)\])? (?<message>.*)$

   [PARSER]
       Name   nginx
       Format regex
       Regex ^(?<remote>[^ ]*) (?<host>[^ ]*) (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^\"]*?)(?: +\S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>[^\"]*)")
       Time_Key time
       Time_Format %d/%b/%Y:%H:%M:%S %z

   [PARSER]
       Name   json
       Format json
       Time_Key time
       Time_Format %d/%b/%Y:%H:%M:%S %z

   [PARSER]
       Name        docker
       Format      json
       Time_Key    time
       Time_Format %Y-%m-%dT%H:%M:%S.%L
       Time_Keep   On

   [PARSER]
         Name cri
         Format regex
         Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
         Time_Key    time
         Time_Format %Y-%m-%dT%H:%M:%S.%L%z
         Time_Keep   On

   [PARSER]
       Name        syslog
       Format      regex
       Regex       ^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$
       Time_Key    time
       Time_Format %b %d %H:%M:%S

`

[patrick-stephens]

Might be worth using the multiline cri,docker parsers that are built in too: https://docs.fluentbit.io/manual/pipeline/inputs/tail#multiline-and-containers-v1.8

[INPUT]
    name              tail
    path              /var/log/containers/*.log
    multiline.parser  docker, cri

[kamaraj-muthupandian]

Hi @patrick-stephens<

Need your inputs/help here

Below is the pod log from two different clusters, in EKS cluster 1.23 log is getting merged but not on EKS cluster 1.24( CRI-O)

On EKS 1.24

2023-07-25T18:36:58.132728019Z stdout F {"collection":"metadata","level":"warning","msg":"No metadata with expiration found","node":"mgmt-5b5fcfb955-qdwxj","pid":1,"service":"mgmt","time":"2023-07-25T18:36:58Z"}

On EKS 1.23

{"log":"{\"level\":\"warning\",\"msg\":\"No metadata with expiration found\",\"node\":\"mgmt-5b5fcfb955-qdwxj\",\"pid\":1,\"service\":\"mgmt\",\"time\":\"2023-07-25T18:36:58Z\",\"time duration\":899999999637}\n","stream":"stdout","time":"2023-07-25T18:36:58.132728019Z"}

[patrick-stephens]

Yeah, you have the CRI format not dockershim is removed.
You need to ensure it is being parsed but it should be by the config you have.

I think the issue is that you are just handling the multiline format, you need a further parser filter to take the log key and parse it into the constituent fields.

The cri filter handles multiline kubelet logs to give you a single log line. Presumably you want to then use a json parser on that final section.

tail (with cri) --> parser (regex to extract json as just a key - maybe decode as JSON) [ --> optional further parser to handle the JSON ]

[kamaraj-muthupandian]

@patrick-stephens Tried with the below filters but it didnt help, not sure I am doing in the right way

[FILTER]
    Name                kubernetes
    Match               kube.*
    Kube_Tag_Prefix     kube.var.log.containers.
    Kube_URL            https://kubernetes.default.svc:443
    Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
    Merge_Log           On
    Merge_Log_Key       log_processed
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On
[FILTER]
    name             parser
    Match            kube.*
    key_name         log
    Parser           json 

[patrick-stephens]

I'm not sure what you're trying to do there, you're telling the K8S filter to merge the logs then use the log_processed key for it but then a subsequent filter tries to JSON parse a different key called log.

You're going to have to provide more details I think on what is happening at each stage and what you intend to happen. You can use the stdout filter (not output) between filters to see the data if you want.

Sample input to reproduce things would be my first goal though as without that it's all just guessing.

[kamaraj-muthupandian]

Hi @patrick-stephens , Below the actual raw string and parsed one . Our end goal is to see the service logs parsed into the log field as proper json

Logs parsed using docker parser EKS 1.23**( Parsed logs are in the expected format)****************

Example:1

Container raw Log:

{"healthcheck_url":"https://test.dnb.com/loc-management/app/v1/locations/status","level":"trace","msg":"checking health","node":"enricher-5f6bd7967b-dnbxt","pid":1,"service":"enricher","service_name":"SDI","time":"2023-08-03T12:08:22Z"}

{"healthcheck_url":"https://test.dnb.com/activate-inv/internal/v1/status","level":"trace","msg":"checking health","node":"enricher-5f6bd7967b-dnbxt","pid":1,"service":"enricher","service_name":"ADI","time":"2023-08-03T12:08:22Z"}

*Parsed logs using docker parser

{"log":"{"healthcheck_url":"https://test.dnb.com/activate-inv/internal/v1/status\",\"level\":\"trace\",\"msg\":\"checking health","node":"enricher-5f6bd7967b-dnbxt","pid":1,"service":"enricher","service_name":"ADI","time":"2023-08-03T12:54:22Z"}\n","stream":"stdout","time":"2023-08-03T12:54:22.027306898Z"}

Example:2

10.188.205.140 - - [03/Aug/2023:12:35:11 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.83.1" "-"

*Parsed logs using docker parser

{"log":"10.188.205.140 - - [03/Aug/2023:12:35:20 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.83.1" "-"\n","stream":"stdout","time":"2023-08-03T12:35:20.737372639Z"}

Logs parsed using cri parser EKS 1.24******************
Example:1

Container raw Log : { Adding some additional timestamp}

{"healthcheck_url":"https://test.hena.com/activate-inv/internal/v1/status","level":"trace","msg":"checking health","node":"enricher-86bd6597b6-lgl7n","pid":1,"service":"enricher","service_name":"ADI","time":"2023-08-03T12:06:59Z"}
2023-08-03T12:06:59.052719986Z {"healthcheck_url":"https://test.hena.com/loc-management/app/v1/locations/status","level":"trace","msg":"checking health","node":"enricher-86bd6597b6-lgl7n","pid":1,"service":"enricher","service_name":"SDI","time":"2023-08-03T12:06:59Z"} 2023-08-03T12:06:59.152095305Z {"duration":99,"level":"trace","msg":"time took to check health in ms","node":"enricher-86bd6597b6-lgl7n","pid":1,"response_code":404,"service":"enricher","service_name":"ADI","service_status":"down","time":"2023-08-03T12:06:59Z"}

*Parsed logs using cri parser

{"healthcheck_url":"https://test.hena.com/activate-inv/internal/v1/status","level":"trace","msg":"checking health","node":"enricher-86bd6597b6-lgl7n","pid":1,"service":"enricher","service_name":"ADI","time":"2023-08-03T13:07:59Z"}

Example:2

10.182.29.128 - - [03/Aug/2023:12:35:09 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.83.1" "-"

*Parsed logs using cri parser

10.182.29.128 - - [03/Aug/2023:12:30:37 +0000] "GET / HTTP/1.1" 200 612 "-" "curl/7.83.1" "-"