Memory usage when extending event buffers is attributed to components

[tobz]

Context

Currently, the global event buffer pool is shared across components in order to move events through a topology. At startup, we allocate a fixed number of these buffers, but crucially, their capacity is empty. As the buffers are used, they are extended as needed.

This causes two main issues:

• event buffers are a potential avenue of unbounded memory growth
• the component that happens to end up causing the buffers to be resized gets stuck with the attributed memory usage

While the growth of these buffers generally won't actually be unbounded -- a lot of other bounds will come into play first -- their misattributed memory usage is problematic, because it can completely blow the budget of a component that otherwise doesn't allocate at all at runtime.

[tobz]

In terms of solutions, this would need to be two-fold:

• event buffers would need to be of fixed-size
• we would need a new object pool that was elastic and could drop unused items after some period of being idle/unused

This would give us the ability to have truly bounded usage -- max pool size * fixed event buffer size -- and to do so in a way that we could apply to the bounds of the topology itself, avoiding components getting hit with the memory usage attribution.

Fixed-size event buffers

This one is the most difficult because while writing such a buffer itself is trivial, using them is less so. Having to add in the code to handle the buffer being full, and then sending it and acquiring a new buffer, would be non-trivial and ugly. At present time, it would be very ugly/contrived with something like the DogStatsD source, based on how we thread through a pre-acquired EventBuffer to Deserializer. Some of the work @lukesteensen is doing might make that a non-issue if we lift up the code where we actually generate, and thus buffer, the events we decoded.

For the rest of the cases, we would potentially write helper adapters to encapsulate the logic of iterating over something that generates the events, and then manages the event buffer being written into, sending it out and acquiring a new one when full.

Elastic object pool

This would be pretty trivial, and would just involve needing to write the implementation.