Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] message field not been parsed with Grok #5247

Closed
Jobsclebson opened this issue Dec 7, 2024 · 1 comment
Closed

[BUG] message field not been parsed with Grok #5247

Jobsclebson opened this issue Dec 7, 2024 · 1 comment
Labels
bug Something isn't working untriaged

Comments

@Jobsclebson
Copy link

Describe the bug
I am ingesting pfSense firewall log (syslog RFC 3164) with Fluetbit:

[SERVICE]
  Flush 1
  Parsers_File parsers.conf

[INPUT]
  Name syslog
  Parser syslog-rfc3164
  Listen 0.0.0.0
  Port 5140
  Mode udp

[OUTPUT]
  Name http
  Match *
  Host data-prepper
  Port 2021
  URI /log/ingest
  HTTP_User admin
  HTTP_Passwd admin

Here is how Fluentbit parses the log before sending to data-prepper:
https://raw.githubusercontent.com/fluent/fluent-bit/refs/heads/master/conf/parsers.conf

PARSER]
    Name        syslog-rfc3164
    Format      regex
    Regex       /^\<(?<pri>[0-9]+)\>(?<time>[^ ]* {1,2}[^ ]* [^ ]*) (?<host>[^ ]*) (?<ident>[a-zA-Z0-9_\/\.\-]*)(?:\[(?<pid>[0-9]+)\])?(?:[^\:]*\:)? *(?<message>.*)$/
    Time_Key    time
    Time_Format %b %d %H:%M:%S
    Time_Keep   On

Here is the data-repper pipeline:

log-pipeline:
  source:
    http:
      ssl: false
  processor:
     - date:
         from_time_received: true
         destination: "@timestamp"
     - grok:
         patterns_directories: ["/usr/share/data-prepper/patterns"]
         match:
           message: ['%{PFSENSE_LOG_ENTRY}']
  sink:
    - opensearch:
        hosts: [ "https://opensearch:9200" ]
        insecure: true
        username: admin
        password: Developer@123
        index: pfsense

Here is the grok patter:
Originaly taken from https://gist.githubusercontent.com/Caligatio/878002ab4aa591747a3dcdbd1101db41/raw/4c0d33b75a6f064dc4b4ae3359fa24d77f2a7fa3/pfsense2-3.grok
I have to made some adjustments because the original was giving me some errors when starting data-prepper.

PFSENSE_LOG_ENTRY %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}?
PFSENSE_LOG_DATA ,%{INT:sub_rule}?,,%{INT:tracker},%{WORD:iface},%{WORD:reason},%{WORD:action},%{WORD:direction},
PFSENSE_IP_SPECIFIC_DATA %{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA}
PFSENSE_IPv4_SPECIFIC_DATA (4:ip_ver),%{BASE16NUM:tos},%{WORD:ecn}?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:proto_id},%{WORD:proto},
PFSENSE_IPv6_SPECIFIC_DATA (6:ip_ver),%{BASE16NUM:class},%{DATA:flow_label},%{INT:hop_limit},%{WORD:proto},%{INT:proto_id},
PFSENSE_IP_DATA %{INT:length},%{IP:src_ip},%{IP:dest_ip},
PFSENSE_PROTOCOL_DATA %{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA}|%{PFSENSE_IGMP_DATA}
PFSENSE_TCP_DATA %{INT:src_port},%{INT:dest_port},%{INT:data_length},%{WORD:tcp_flags},%{INT:sequence_number},%{INT:ack_number},%{INT:tcp_window},%{DATA:urg_data},%{DATA:tcp_options}
PFSENSE_UDP_DATA %{INT:src_port},%{INT:dest_port},%{INT:data_length}
PFSENSE_IGMP_DATA datalength=%{INT:data_length}
PFSENSE_ICMP_DATA %{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE}
PFSENSE_ICMP_TYPE ((request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply):imcp_type),
PFSENSE_ICMP_RESPONSE %{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY}
PFSENSE_ICMP_ECHO_REQ_REPLY %{INT:icmp_echo_id},%{INT:icmp_echo_sequence}
PFSENSE_ICMP_UNREACHPORT %{IP:icmp_unreachport_dest_ip},%{WORD:icmp_unreachport_protocol},%{INT:icmp_unreachport_port}
PFSENSE_ICMP_UNREACHPROTO %{IP:icmp_unreach_dest_ip},%{WORD:icmp_unreachproto_protocol}
PFSENSE_ICMP_UNREACHABLE %{GREEDYDATA:icmp_unreachable}
PFSENSE_ICMP_NEED_FLAG %{IP:icmp_need_flag_ip},%{INT:icmp_need_flag_mtu}
PFSENSE_ICMP_TSTAMP %{INT:icmp_tstamp_id},%{INT:icmp_tstamp_sequence}
PFSENSE_ICMP_TSTAMP_REPLY %{INT:icmp_tstamp_reply_id},%{INT:icmp_tstamp_reply_sequence},%{INT:icmp_tstamp_reply_otime},%{INT:icmp_tstamp_reply_rtime},%{INT:icmp_tstamp_reply_ttime}
PFSENSE_CARP_DATA %{WORD:carp_type},%{INT:carp_ttl},%{INT:carp_vhid},%{INT:carp_version},%{INT:carp_advbase},%{INT:carp_advskew}

Here is what I get on opensearch:
image
The field message is not been parsed.

To Reproduce
Steps to reproduce the behavior:

  1. Using the sample from https://github.com/opensearch-project/data-prepper/blob/main/examples/log-ingestion/README.md
  2. And make the adjustments above.

Expected behavior
I expected the message part of the log been splinted by fields as specified on the grok pattern.

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

Additional context
Add any other context about the problem here.

@Jobsclebson
Copy link
Author

I changed the lines bellow:

PFSENSE_IPv4_SPECIFIC_DATA (4:ip_ver),%{BASE16NUM:tos},%{WORD:ecn}?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:proto_id},%{WORD:proto},
PFSENSE_IPv6_SPECIFIC_DATA (6:ip_ver),%{BASE16NUM:class},%{DATA:flow_label},%{INT:hop_limit},%{WORD:proto},%{INT:proto_id},
PFSENSE_ICMP_DATA %{PFSENSE_ICMP_TYPE}%{PFSENSE_ICMP_RESPONSE}
PFSENSE_ICMP_TYPE ((request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply):imcp_type),

This is working:

PFSENSE_LOG_ENTRY %{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}?
PFSENSE_LOG_DATA %{INT:rule}?,%{INT:sub_rule}?,,%{INT:tracker},%{WORD:iface},%{WORD:reason},%{WORD:action},%{WORD:direction},
PFSENSE_IP_SPECIFIC_DATA %{PFSENSE_IPv4_SPECIFIC_DATA}|%{PFSENSE_IPv6_SPECIFIC_DATA}
PFSENSE_IPv4_SPECIFIC_DATA %{INT:ip_ver},%{BASE16NUM:tos},%{WORD:ecn}?,%{INT:ttl},%{INT:id},%{INT:offset},%{WORD:flags},%{INT:proto_id},%{WORD:proto},
PFSENSE_IPv6_SPECIFIC_DATA %{INT:ip_ver},%{BASE16NUM:class},%{DATA:flow_label},%{INT:hop_limit},%{WORD:proto},%{INT:proto_id},
PFSENSE_IP_DATA %{INT:length},%{IP:src_ip},%{IP:dest_ip},
PFSENSE_PROTOCOL_DATA %{PFSENSE_TCP_DATA}|%{PFSENSE_UDP_DATA}|%{PFSENSE_ICMP_DATA}|%{PFSENSE_CARP_DATA}|%{PFSENSE_IGMP_DATA}
PFSENSE_TCP_DATA %{INT:src_port},%{INT:dest_port},%{INT:data_length},%{WORD:tcp_flags},%{INT:sequence_number},%{INT:ack_number},%{INT:tcp_window},%{DATA:urg_data},%{DATA:tcp_options}
PFSENSE_UDP_DATA %{INT:src_port},%{INT:dest_port},%{INT:data_length}
PFSENSE_IGMP_DATA datalength=%{INT:data_length}
PFSENSE_ICMP_DATA %{PFSENSE_ICMP_TYPE:imcp_type},%{PFSENSE_ICMP_RESPONSE}
PFSENSE_ICMP_TYPE (request|reply|unreachproto|unreachport|unreach|timeexceed|paramprob|redirect|maskreply|needfrag|tstamp|tstampreply)
PFSENSE_ICMP_RESPONSE %{PFSENSE_ICMP_ECHO_REQ_REPLY}|%{PFSENSE_ICMP_UNREACHPORT}| %{PFSENSE_ICMP_UNREACHPROTO}|%{PFSENSE_ICMP_UNREACHABLE}|%{PFSENSE_ICMP_NEED_FLAG}|%{PFSENSE_ICMP_TSTAMP}|%{PFSENSE_ICMP_TSTAMP_REPLY}
PFSENSE_ICMP_ECHO_REQ_REPLY %{INT:icmp_echo_id},%{INT:icmp_echo_sequence}
PFSENSE_ICMP_UNREACHPORT %{IP:icmp_unreachport_dest_ip},%{WORD:icmp_unreachport_protocol},%{INT:icmp_unreachport_port}
PFSENSE_ICMP_UNREACHPROTO %{IP:icmp_unreach_dest_ip},%{WORD:icmp_unreachproto_protocol}
PFSENSE_ICMP_UNREACHABLE %{GREEDYDATA:icmp_unreachable}
PFSENSE_ICMP_NEED_FLAG %{IP:icmp_need_flag_ip},%{INT:icmp_need_flag_mtu}
PFSENSE_ICMP_TSTAMP %{INT:icmp_tstamp_id},%{INT:icmp_tstamp_sequence}
PFSENSE_ICMP_TSTAMP_REPLY %{INT:icmp_tstamp_reply_id},%{INT:icmp_tstamp_reply_sequence},%{INT:icmp_tstamp_reply_otime},%{INT:icmp_tstamp_reply_rtime},%{INT:icmp_tstamp_reply_ttime}
PFSENSE_CARP_DATA %{WORD:carp_type},%{INT:carp_ttl},%{INT:carp_vhid},%{INT:carp_version},%{INT:carp_advbase},%{INT:carp_advskew}

image

@github-project-automation github-project-automation bot moved this from Unplanned to Done in Data Prepper Tracking Board Dec 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working untriaged
Projects
Development

No branches or pull requests

1 participant