How to alter the recordHandler() function to process the entire batch of events at once rather than one at a time?

[msbeeman]

Hey everyone,

I had a question regarding the batch tools, specifically on how to handle processing the records in batches rather than one at a time. The docs seems unclear how I should handle batch processing the DynamoDBRecords the lambda receives (or even if I can do such a thing?). It seems like the recordHandler() function only accepts and processes one DynamoDBRecord at a time. What if I, for example, want to map over all the records at once so I can batch the processed records into a single dynamodb batch write request (which accepts up to 25 puts at a time)? Does anyone know how I could handle that if the recordHandler can only receive and process one record at a time and is then called directly in the return function for each individual event record?

[dreamorosi]

Hi, as mentioned in #1658 and on Discord this is not possible at the moment.

The Batch Processing utility primarily goal is to help you process records and handle partial failures. To do this, we have designed the utility so that the record handler function that you provide us is called once for each item.

If you only want to create smaller chunks of a batch and write to DynamoDB the content of these chunks, perhaps you don't need the utility at all and you could handle this in the Lambda handler directly.

You can however extend the BatchProcessor.successHandler() and BatchProcessor.failureHandler() methods to keep track of the number of records processed and optionally issue a DynamoDB batch write when the number reaches a certain amount.

I would suggest you do this only if you're using the Batch Processing utility to do something with the record handler on each record and also need the partial failures feature; if not I'd simply split the event.Records array in smaller chunks and do the writes directly like shown in my comment here.