fix(transform): Change Kinesis Payload Limit to MB #206
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
Discovered an edge case where the system generated a Kinesis Data Stream payload of more than 5 MB when using the PutRecords API. Changing the limit of each record from 1 MiB (1048576 bytes) to 1 MB (1000000 bytes) fixed the problem. According to the AWS docs, Firehose uses 1 MiB as its message size limit.
How Has This Been Tested?
Tested E2E in the production data pipeline the bug was discovered in.
Types of changes
Checklist: