Use better default values for workers/pools/buffers

[jordansissel]

Things to tune and/or make default:

• workers, based on number of CPU cores
• review pool/buffer sizes w/ tests for performance
• Change default protocol in ES output from node to http RFC: Change the default protocol to http logstash-plugins/logstash-output-elasticsearch#150
• Audit existing defaults for Kafka consumer Audit existing defaults for Kafka consumer logstash-plugins/logstash-input-kafka#36
• Change the default bulk size to 500 instead of 5000 documents Change the default bulk size to 500 instead of 5000 documents logstash-plugins/logstash-output-elasticsearch#181
• Remove original/temporary timestamp field by default Remove original/temporary timestamp field by default logstash-plugins/logstash-filter-date#24

Specific to plugins:

• redis: use batch operations by default.
• enable batch operations by default where possible

Specific Issues:

Will be updated as we file more individual issues against plugins

[kobazik]

+1

[jordansissel]

Wanted to use Elasticsearch's bundled Sigar libs, but the build process removes them. Oldest commit I can find for why I did this is here: c78d1e3 and I haven't a clue why I did it ;)

Easy to fix. ;)

[colinsurprenant]

🎶 Come in here, dear boy, have a Sigar.
You're gonna go far, you're gonna fly high,
You're never gonna die, you're gonna make it if you try; they're gonna love you. 🎶

[jlintz]

@jordansissel given that some filters aren't thread safe (multiline), I wonder if it makes sense to check if any such filters are configured before setting the workers = # cpu cores?

[jordansissel]

Indeed! Part of this work will be to detect that and act accordingly.

[guyboertje]

I want to begin a discussion about which 'values' are considered default and important.

Please refer to this overview meta issue

[guyboertje]

pipeline.rb

• line 36 SizedQueue.new(20)
• line 44 @settings = { "filter-workers" => 1 }
• line 165 Thread.new { Stud.interval(5) { @input_to_filter.push(LogStash::FLUSH) } }

Action for line 44: Add support for filter_workers default to cpu-cores

[guyboertje]

agent.rb

• line 22 option ["-w", "--filterworkers"], "COUNT", default = 1
• line 26 option "--watchdog-timeout", "SECONDS", default = 10
• line 131 Thread.new(@logger) {|logger| sleep 5; logger.warn(I18n.t("logstash.agent.slow_shutdown"))

Action for line 22: Add support for filter_workers default to cpu-cores

Action for line 26: Remove Watchdog in the code base

[guyboertje]

/outputs/base.rb

• line 71 @worker_queue = SizedQueue.new(20)

[suyog] I would leave this as-is

[ph]

The current implementation of the watchdog will go alway, I'll make a PR to remove it completely.
It's not currently used and it was a source of problems for some plugins DNS IIRC.

[suyograo]

@guyboertje @settings = { "filter-workers" => 1 } is definitely candidate for changing. IMO we should default that to # of cores in the system. But we need to be careful of one usecase: multiline filter which needs events to be in order (and will not work with -w). But, we can just check for existence of that filter at start up and not use # of cores

[guyboertje]

@suyograo - there are checks for in-order events AFAIK.

[guyboertje]

inputs/elasticsearch.rb

• line 50 config :size, :validate => :number, :default => 1000
• line 55 config :scroll, :validate => :string, :default => "1m"

[guyboertje]

inputs/http.rb

• line 50 config :threads, :validate => :number, :default => 4 should be no. of cpus

[guyboertje]

inputs/imap.rb

• line 26 config :fetch_count, :validate => :number, :default => 50
• line 28 config :check_interval, :validate => :number, :default => 300

[guyboertje]

inputs/kafka.rb

• line 56 config :consumer_threads, :validate => :number, :default => 1 should this be called 'partitions'?
• line 58 config :queue_size, :validate => :number, :default => 20
• line 63 config :rebalance_max_retries, :validate => :number, :default => 4
• line 65 config :rebalance_backoff_ms, :validate => :number, :default => 2000

[guyboertje]

inputs/lumberjack.rb

• line 40 BUFFERED_QUEUE_SIZE = 1
• line 41 RECONNECT_BACKOFF_SLEEP = 0.5

[guyboertje]

inputs/rabbitmq.rb

• line 76 config :prefetch_count, :validate => :number, :default => 256

[guyboertje]

inputs/redis.rb

• line 59 config :batch_count, :validate => :number, :default => 1 should this start at queue size (20) and, possibly, dynamically change to grab as many as would fill the queue?

Action for line 59

[guyboertje]

NOTE: Magic numbers/strings, just convert the magic value into a descriptive method call from LogStash::Util::Defaults

[purbon]

done the timestamp behaviour at -> logstash-plugins/logstash-filter-date#36