Add standard chunking format #914
Conversation
* to enable limits by # of records for buffer chunks * to be able to visualize data flow in fluentd
* it improves performance * it's needed to implement output plugins like ObjectBufferedOutput of v0.12
* This type of output provides the standard format of binary representation of buffer chunk payloads * It is actually equal to results of EventStream#to_msgpack_stream * We can provide some tools to read-and-rescue buffer chunks in failure situations for such chunks * This output style is actually same with ObjectBufferedOutput of v0.12
|
This change is basis for many features in future.
|
|
@repeatedly could you review this? |
| @@ -21,6 +21,10 @@ class EventStream | |||
| include Enumerable | |||
| include MessagePackFactory::Mixin | |||
|
|
|||
| def records | |||
There was a problem hiding this comment.
The name records seems to represent returns the event entries.
num_records or size, rubish, is better for me.
There was a problem hiding this comment.
size looks bad, because it sometimes means bytes of a data, and a type of EventStream (MessagePackEventStream) is actually a String object.
How about events?
There was a problem hiding this comment.
events is better for me.
@sonots How about this? This method will be used in flowcounter_simple or similar gems.
There was a problem hiding this comment.
It looks this method returns an integer, not an array. So, both records and events look bad for me.
What about num_events, num_records, or just length?
There was a problem hiding this comment.
Once I would use length, but I re-found that Chunk#records already exists, and I made EventStream#records as same name with it.
Chunk also has Chunk#size (bytesize), and #size != #length is extraordinary confusing in ruby way. Hmm.
There was a problem hiding this comment.
I think it's best to:
- change
Chunk#sizeto return # of events, and make#lengthas alias of#size - add
Chunk#bytesizeto return # of bytes of chunk content
But this changes the meaning of chunk.size, and it will break compatibility of plugins.
There was a problem hiding this comment.
I'll create another pull-request to change Chunk#records and EventStream#records to #num_events later.
Let me merge this right now.
This change is to add formatting for buffer chunks. Output without explicit
#formatdefinition will use this standard format to write event streams to buffer chunks, and actually, it's same withEventStream.to_msgpack_stream.At the same time, this change moves a method to iterate events from chunks to
event.rbas mixin. It's about event streams, not chunk itself.