Provide option to have variable size internal queues which are persisted

[suyograo]

As mentioned in #2605, Logstash uses in-memory bounded queues between pipeline stages (input to filter, filter to output) to buffer events. The size of the queue is restricted to 20 and is non-configurable. This works well in practice because:

1. Provides efficient streaming of messages between stages to reduce overall latency
2. Not so large as to cause memory pressure and GC activity
3. Not so large that crashing would cause devastating loss of events (by count). See Add queue persistence to make pipeline resilient to crashes and restarts #2605

In Logstash deployments which require high throughput and resiliency, users typically deploy message brokers such as Redis, RabbitMQ or Apache Kafka. Essentially, this breaks the pipeline into two stages:

1. Shipping stage: The application will ship logs to a local Logstash instance without any filters. This Logstash instance will forward events to an external message broker to buffer them
2. Indexing/Filter stage: A second group of Logstash instances will pull off the queue, apply the filters, and output to destinations like Elasticsearch, Pager Duty, Amazon S3.

This helps cases when there is a mismatch in cadence between the shipping and the relatively expensive processing stage. The queue buffers events outside of the application machines and does not add back pressure to the source.

Building on the work of persistent queues (#1939), we plan to offer a built-in alternative to an external message broker. By adding a variable queueing option to Logstash which is persisted to disk, we will provide an option to remove the dependency on external queues. This will make it operationally easier to deploy and maintain Logstash instances. We will provide APIs to monitor and interact with the queues.

[pemontto]

👍

[bitsofinfo]

Ditto'ing the comment on #2605, it would be great if this built in buffering mechanism was based on an abstracted contract which could be implemented via an extensible framework through plugins

[suyograo]

We will be closing this as the first iteration of Persistent Queues in Logstash will be variable length. Follow progress in #5638