awslabs#38 - Deflate messages before checking if record is within max…

… data size limit. Added documentation in README for gzip_compression

README.md

@@ -203,6 +203,12 @@ fails all retries an error log will be emitted.
 Boolean, default is false.
 In case you find error `Encoding::UndefinedConversionError` with multibyte texts, you can avoid that error with this option.
 
+### gzip_compression
+
+Boolean, default is false.
+Gzip compresses the message data blob.
+Each gzip compressed message must remain within megabyte in size.
+
 ### debug
 
 Boolean. Enable if you need to debug Amazon Kinesis API call. Default is false.

lib/fluent/plugin/out_kinesis.rb

@@ -122,15 +122,15 @@ def format(tag, time, record)
     end
 
     def write(chunk)
-      data_list = chunk.to_enum(:msgpack_each).find_all{|record|
-        unless record_exceeds_max_size?(record['data'])
+      data_list = chunk.to_enum(:msgpack_each).map{|record|
+        build_data_to_put(record)
+      }.find_all{|record|
+        unless record_exceeds_max_size?(record[:data])
           true
         else
-          log.error sprintf('Record exceeds the %.3f KB(s) per-record size limit and will not be delivered: %s', PUT_RECORD_MAX_DATA_SIZE / 1024.0, record['data'])
+          log.error sprintf('Record exceeds the %.3f KB(s) per-record size limit and will not be delivered: %s', PUT_RECORD_MAX_DATA_SIZE / 1024.0, record[:data])
           false
         end
-      }.map{|record|
-        build_data_to_put(record)
       }
 
       if @order_events