json keys that look like references get dereferenced

[bazzargh]

• Version: logstash-filter-json (3.0.5) (as part of logstash 6.5.1)
• Operating System: osx 10.13.6
• Config File (if you have sensitive info, please remove it):

output {
  stdout {
    codec => rubydebug
  }
}
input {
  generator {
    lines => [
      '{"foo":"bar","nested":{"one":"two"},"not_nested[\"three\"]":"four","[what][is][this]":"five"}'
    ]
    count => 1
  }
}
filter {
  json {
    source => "message"
    remove_field => ["message"]
  }
}

• Sample Data: in conf above
• Steps to Reproduce: logstash -f bug.conf

Expected:

{
      "sequence" => 0,
    "@timestamp" => 2018-12-05T18:28:29.623Z,
      "[what][is][this]" => "five",
      "@version" => "1",
          "host" => "laptop.local",
      "not_nested[\"three\"]" => "four",
           "foo" => "bar",
        "nested" => {
        "one" => "two"
    }
}

Actual:

{
      "sequence" => 0,
    "@timestamp" => 2018-12-05T18:28:29.623Z,
          "what" => {
        "is" => {
            "this" => "five"
        }
    },
      "@version" => "1",
          "host" => "laptop.local",
    "not_nested" => {
        "\"three\"" => "four"
    },
           "foo" => "bar",
        "nested" => {
        "one" => "two"
    }
}

We spotted this with log data where a query string had ref-like keys, and got the "Detected ambiguous Field Reference" warning. I was surprised that it's doing anything with this at all, and there doesn't seem to be an option to disable this behaviour - these are keys in our log data not conf for processing.

[TheVastyDeep]

Similarly,

input { generator { count => 1 lines => [ '{ "foo[1]": "bar" }' ] } }
filter { json { source => "message" } }

will cause logstash to crash

RuntimeError: Invalid FieldReference: `foo[1]`
        set at org/logstash/ext/JrubyEventExtLibrary.java:92
     filter at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-json-3.0.6/lib/logstash/filters/json.rb:106
       each at org/jruby/RubyHash.java:1419
     filter at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-json-3.0.6/lib/logstash/filters/json.rb:106
  do_filter at /usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143

or

org.jruby.exceptions.RuntimeError: (RuntimeError) Invalid FieldReference: `foo[1]`
    at org.logstash.ext.JrubyEventExtLibrary$RubyEvent.set(org/logstash/ext/JrubyEventExtLibrary.java:92) ~[logstash-core.jar:?]
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_json_minus_3_dot_0_dot_6.lib.logstash.filters.json.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-json-3.0.6/lib/logstash/filters/json.rb:106) ~[?:?]
    at org.jruby.RubyHash.each(org/jruby/RubyHash.java:1419) ~[jruby-complete-9.2.7.0.jar:?]
    at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_filter_minus_json_minus_3_dot_0_dot_6.lib.logstash.filters.json.filter(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-json-3.0.6/lib/logstash/filters/json.rb:106) ~[?:?]
    at usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143) ~[?:?]

depending on whether you have java_execution enabled.

[IngaFeick]

I am observing the same issue that @TheVastyDeep describes.

[mingyuan-xia]

+1 for this headache.
The root cause is https://github.com/logstash-plugins/logstash-filter-json/blob/master/lib/logstash/filters/json.rb#L106
where event.set will dereference JSON keys. Keys with brackets will have unexpected outcomings. For example: [] crashes, [non_existing_field] resolves to null, [welcome[]] crashes.
Currently, we take a workaround by exploiting https://github.com/logstash-plugins/logstash-filter-json/blob/master/lib/logstash/filters/json.rb#L86
With a target set, the parsed JSON will be put into the target, free from the dereferencing bug.

[chenchuangc]

is this problem solved today ?, i meet the same problem in 7.5.5

[chenchuangc]

+1 for this headache. The root cause is https://github.com/logstash-plugins/logstash-filter-json/blob/master/lib/logstash/filters/json.rb#L106 where event.set will dereference JSON keys. Keys with brackets will have unexpected outcomings. For example: [] crashes, [non_existing_field] resolves to null, [welcome[]] crashes. Currently, we take a workaround by exploiting https://github.com/logstash-plugins/logstash-filter-json/blob/master/lib/logstash/filters/json.rb#L86 With a target set, the parsed JSON will be put into the target, free from the dereferencing bug.

can you help me show the detail?
i use

json {
              source => "message_log"
              target => "doc.custom"
              remove_field => ["message","message_log"]
          }

but still hold a lot of memory

[qaiserali]

Hi,

I'm getting the same issue with my logstash. Any idea how to fix this?