Fluentd filter plugin to split a record into multiple records with key/value pair. Compatible with 0.12 and 0.14 versions of fluentd.
This plugin splits a record and parses each results to make key/value pairs; Logstash's kv filter is a good example of this. It is a successor to fluent-plugin-split; a 0.10 output plugin. This is NOT the current 0.12+ fluent-plugin-split: that one is what currently installs with ruby-gem, and splits CSV-style content.
Normally you can use a regular expression to parse a record. It is difficult to parse a record which has ambiguous numbers of data like a following record.
Before
{"message":"key1=val1 key2=val2 key3=val3"}After
{"key1":"val1","key2":"val2","key3":"val3"}$ git clone https://github.com/unquietwiki/fluent-plugin-split_record.git && cd fluent-plugin-split_record
$ td-agent-gem build fluent-plugin-split_record.gemspec
$ td-agent-gem install fluent-plugin-split_record-0.12.1.gem
$ td-agent-gem install fluent-plugin-split_record
| parameter | description | default |
|---|---|---|
| tag | key name for tag | |
| format | regexp to parse a record after split | '(?\S*)=(?\S*)' |
| substring_format | regexp used to identify substrings | '(?\S*)=\"(?.*?)\"' |
| key_name | key name to be split | |
| out_key | key name of json object which includes divided records | nil |
| reserve_msg | if original message is reserved or not | nil |
| keys_prefix | if set, all extracted keys names will be preceded by this string | nil |
You may want to pre-process with the regexp parser to remove/tag other elements first; this is a requirement if working with SonicWall syslog input, which is otherwise an array of key-value pairs.
<source>
@type udp
port 514
format /\<(?<prefix>[0-9]{1,3})\>(?<extradata>.+)$\z/
tag FW
</source>
<filter FW.**>
@type split_record
tag FW
key_name extradata
reserve_msg no
</filter>