What is the consequence of setting batchTimeOut to 15 seconds

[tweng100]

Hi,
I read the documentation, the recommended setting for batchTimeout is over 2 minutes. Do you know what is the consequence of setting the timeout to 15 seconds? what are the drawbacks?

Thanks,
Tao

[IanMeyers]

Such a short timeout will likely result in multiple loads at the same time, which can create commit contention and cause other transactions to be delayed.

[taoweng100]

Thank you very much for your reply. This is really helpful.
If I change the lambda function by putting all redshift copy requests into a queue, and create one receiver that polls the copy requests from the queue one by one and execute them continuously in sequence, could this workaround the 2 minutes limitation? and achieve realtime data delivery? any drawback with this approach?

[IanMeyers]

In theory you could do that. Alternatively, simply write a task that loads all of the copy commands that you’ve run from the Batch history table, and then set the time out to that value +1/ second. In that way you will balance the load time vs commit contention.

[taoweng100]

thank you very much for the reply. I spent the last couple days learning s3 redshift loader code and still have trouble understand the alternative approach. If possible, could you please elaborate with more details. :-)