How to buffer records? #497
Replies: 13 comments 7 replies
-
Hi @todor-kolev. There can be up to 24 (configurable) This mechanism is designed to get a low latency (buffer time) as low as possible, while having high throughput (batch size), depending on the time it takes to send Although I don't know exactly how the AWS Kinesis Producer library (KPL) does it, based on the config docs, it will always buffer records for some time. Hope that helps, let me know if I can explain further. |
Beta Was this translation helpful? Give feedback.
-
Have you enabled the For a high production rate, I would expect that batch size would be maximised. There should be log lines like so:
Can you get an idea of the average batch size? |
Beta Was this translation helpful? Give feedback.
-
We've enabled aggregation, yep. The majority of log statements have batch
size of 7-8 and 1 aggregated eg:
PutRecords for batch of size 8 (1 aggregated)
…On Fri, 30 Apr 2021, 18:07 svroonland, ***@***.***> wrote:
Have you enabled the aggregate setting in the ProducerSettings to
aggregate multiple records into one? This is an additional level of
aggregation, on top of batching. KPL does this as well, enabled by default
I believe, but I see it's disabled by default in zio-kinesis.
For a high production rate, I would expect that batch size would be
maximised. There should be log lines like so:
PutRecords for batch of size 500 (500 aggregated)
Can you get an idea of the average batch size?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#497 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACWFCVCYC4W3DPLD3YXYII3TLLPVZANCNFSM433Z4L7A>
.
|
Beta Was this translation helpful? Give feedback.
-
I just ran some tests and I noticed that the batching behavior is indeed not as intended. With some changes to buffer sizes, I'm seeing a lot more batches at 500 or close to it and a lot higher throughput. I'll try to release a patch as soon as possible. |
Beta Was this translation helpful? Give feedback.
-
The speedup is rather embarrassingly good actually! I'm not sure if this is a regression or if I overlooked this critical part during initial development. Could you try the latest snapshot release? https://oss.sonatype.org/content/repositories/snapshots/nl/vroste/zio-kinesis_2.13/0.18.2+22-8134bfff-SNAPSHOT/ |
Beta Was this translation helpful? Give feedback.
-
So that's about 271 kinesis records per second of about 50 Kb each. With that load it's possible that the outgoing The member length issue looks like a bug in the batching or aggregation size calculation, I'll look into that. That could cause lower throughput, since those records will be retried a couple of times but always fail (I think). A few additional questions:
|
Beta Was this translation helpful? Give feedback.
-
Alright, so about 9 kinesis records per shard per second then, right? With max 24 PutRecords calls in parallel (the default for To get back to your question about adding an option to always buffer for a fixed period of time: not that I'm not willing to consider it, but I don't think it should be necessary, if everything works as designed. The Producer should find the optimal batch sizes, depending on the load and the network speed. It should not be a limiting factor to the throughput. The latest snapshot release contains a fix for the record size issue (#499) + another performance improvement I had in the works (#501). Perhaps you could check CPU load and throughput again with the latest version? |
Beta Was this translation helpful? Give feedback.
-
What snapshot version is that exactly? |
Beta Was this translation helpful? Give feedback.
-
That's the correct one. Just to double check. I'd have to look further into the member length issue. May take a couple of days, but I will keep you posted. |
Beta Was this translation helpful? Give feedback.
-
I'm able to reproduce it by running with random record sizes for a long time. Now to find the exact cause. |
Beta Was this translation helpful? Give feedback.
-
The size estimation of the protobuf-aggregated records was too small, leading it sometimes to over the 1 MB limit. For now I limited it at 90% of the limit, until I figure out the exact size calculation based on the protobuf spec or work on a way to retry correctly when the limit is hit. This should at least prevent the issue from occurring for now. It's in the latest snapshot, looking forward to your feedback. |
Beta Was this translation helpful? Give feedback.
-
Thanks, good to hear the error is gone. There is probably still a lot of optimization that can be done for the Producer, but the JVM implementation is likely less performant than the KPL which is written in C++. To see if a backlog is building up, you could print out the metrics and look whether the latency is increasing. |
Beta Was this translation helpful? Give feedback.
-
Some more performance improvements in the latest snapshot. CPU usage and garbage collection has reduced quite significantly. |
Beta Was this translation helpful? Give feedback.
-
A comment in ProducerLive#runloop says that it will "Batch records up to the Kinesis PutRecords request limits as long as downstream is busy". Does that mean that if downstream is not busy records won't be batched? Is there a way to buffer records like you would with
KinesisProducerConfiguration#setRecordMaxBufferedTime
for example?Beta Was this translation helpful? Give feedback.
All reactions