fluent-plugin-record-reformer

Fluentd plugin to add or replace fields of a event record

Requirements

See .travis.yml

Note that fluent-plugin-record-reformer supports both v0.14 API and v0.12 API in one gem.

Installation

Use RubyGems:

gem install fluent-plugin-record-reformer

Configuration

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  
  tag reformed.${tag_prefix[-2]}
  <record>
    hostname ${hostname}
    input_tag ${tag}
    last_tag ${tag_parts[-1]}
    message ${record['message']}, yay!
  </record>
</match>

Assume following input is coming (indented):

foo.bar {
  "remove_me":"bar",
  "not_remove_me":"bar",
  "message":"Hello world!"
}

then output becomes as below (indented):

reformed.foo {
  "not_remove_me":"bar",
  "hostname":"YOUR_HOSTNAME",
  "input_tag":"foo.bar",
  "last_tag":"bar",
  "message":"Hello world!, yay!",
}

Configuration (Classic Style)

Example:

<match foo.**>
  type record_reformer
  remove_keys remove_me
  renew_record false
  enable_ruby false
  tag reformed.${tag_prefix[-2]}
  
  hostname ${hostname}
  input_tag ${tag}
  last_tag ${tag_parts[-1]}
  message ${record['message']}, yay!
</match>

This results in same, but please note that following option parameters are reserved, so can not be used as a record key.

Option Parameters

• output_tag (obsolete)

  The output tag name. This option is deprecated. Use tag option instead

• tag

  The output tag name.

• remove_keys

  Specify record keys to be removed by a string separated by , (comma) like

    remove_keys message,foo
  

• renew_record bool

  renew_record true creates an output record newly without extending (merging) the input record fields. Default is false.

• renew_time_key string

  renew_time_key foo overwrites the time of events with a value of the record field foo if exists. The value of foo must be a unix time.

• keep_keys

  You may want to remain some record fields although you specify renew_record true. Then, specify record keys to be kept by a string separated by , (comma) like

    keep_keys message,foo
  

• enable_ruby bool

  Enable to use ruby codes in placeholders. See Placeholders section. Default is true (just for lower version compatibility).

• auto_typecast bool

  Automatically cast the field types. Default is false. NOTE: This option is effective only for field values comprised of a single placeholder.

  Effective Examples:

    foo ${foo}
  

  Non-Effective Examples:

    foo ${foo}${bar}
    foo ${foo}bar
    foo 1
  

  Internally, this keeps the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.

  When you need to cast field types manually, out_typecast and filter_typecast are available.

Placeholders

Following placeholders are available:

• ${record["key"]} Record value of key such as ${record["message"]} in the above example (available from v0.8.0).

  • Originally, record placeholders were available as ${key} such as ${message}. This is still kept for the backward compatibility, but would be removed in the future.

• ${hostname} Hostname of the running machine

• ${tag} Input tag

• ${time} Time of the event

• ${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'

• ${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}, ${tag_parts[-1]}.

• ${tag_prefix[N]} Tag parts before and on the index N. For example,

    Input tag: prefix.test.tag.suffix
    
    ${tag_prefix[0]}  => prefix
    ${tag_prefix[1]}  => prefix.test
    ${tag_prefix[-2]} => prefix.test.tag
    ${tag_prefix[-1]} => prefix.test.tag.suffix
  

• ${tag_suffix[N]} Tag parts after and on the index N. For example,

    Input tag: prefix.test.tag.suffix
  
    ${tag_suffix[0]}  => prefix.test.tag.suffix
    ${tag_suffix[1]}  => test.tag.suffix
    ${tag_suffix[-2]} => tag.suffix
    ${tag_suffix[-1]} => suffix
  

It is also possible to write a ruby code in placeholders if you set enable_ruby true option, so you may write some codes as

• ${time.strftime('%Y-%m-%dT%H:%M:%S%z')}
• ${tag_parts.last}

but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.

Relatives

Following plugins look similar:

• fluent-plugin-record-modifier
• fluent-plugin-format
• fluent-plugin-add
• filter_record_transformer is a Fluentd v0.12 built-in plugin which is based on record-reformer.

ChangeLog

See CHANGELOG.md for details.

Contributing

1. Fork it
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create new Pull Request

Copyright

Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.