Java Arraylists do not play well with beefcake

[jhitze]

Logstash seems to be sending [tags] as a Java ArrayList. Beefcake tries to do a 'compact' on it, assuming it's a Ruby Array, and throws an unknown method exception.

/opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:114:in `encode!',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:103:in `encode',
 org/jruby/RubyArray.java:1613:in `each',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:96:in `encode',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:118:in `encode!',
 org/jruby/RubyArray.java:1613:in `each',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:114:in `encode!',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:103:in `encode',
 org/jruby/RubyArray.java:1613:in `each',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/beefcake-1.1.0/lib/beefcake.rb:96:in `encode',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/message.rb:13:in `encode_with_length',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/client/tcp.rb:51:in `send_recv',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/client/tcp.rb:65:in `with_connection',
 org/jruby/ext/thread/Mutex.java:149:in `synchronize',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/client/tcp.rb:62:in `with_connection',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/client/tcp.rb:50:in `send_recv',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/riemann-client-0.2.1/lib/riemann/client.rb:47:in `<<',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-riemann-0.2.0/lib/logstash/outputs/riemann.rb:161:in `send_to_riemann',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-riemann-0.2.0/lib/logstash/outputs/riemann.rb:130:in `receive',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/outputs/base.rb:88:in `handle',
 (eval):798:in `output_func',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:244:in `outputworker',
 /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/pipeline.rb:166:in `start_outputs'

[jordansissel]

Can you attach your logstash config?

@colinsurprenant I think this is another isntance of JrJackson giving us ArrayList and we aren't converting fully.

[suyograo]

/cc @guyboertje. Can you please take a look? Similar issue here: elastic/logstash#3482

[jhitze]

Relevant config portions:

input {
  redis {
    host => "127.0.0.1"
    type => "redis-input"
    threads => 4
    # these settings should match the output of the agent
    data_type => "list"
    key => "logstash"

    # We use json_event here since the sender is a logstash agent
    codec => "json"
  }
}

output {
  riemann {
    host => "logstash.purduefed.com"
    map_fields => true
  }
  elasticsearch {
    cluster => "es_log_cluster"
    protocol => "transport"
    host => ["127.0.0.1"]
  }
}

[jhitze]

We (well, @McPolemic) have a hack to get around it.

 public
  def map_fields(parent, fields)
    this_level = Hash.new
    fields.each do |key, contents|
      next if key.start_with?("@")
      field = parent.nil? ? key : "#{parent}.#{key}"
      if contents.is_a?(Hash)
        this_level.merge! map_fields(field, contents)
      # begin hack
      elsif contents.is_a?(Array)
        this_level[field.to_sym] = contents.to_a
      # end hack
      else
        this_level[field.to_sym] = contents
      end
    end
    return this_level
  end

Once we do that, we run into a different issue where beefcake tries to compare a String to Int64. We did some preliminary debugging, and it only happened on fields with a name of time. Strange stuff.

[guyboertje]

PR coming soon

[guyboertje]

PR is in main logstash repo - elastic/logstash#3772

[guyboertje]

PR merged closing