fix(transform): Empty Aggregated Kinesis Records #171
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Motivation and Context
This is a fix that is in addition to #168.
There's an edge case where an empty aggregated Kinesis record can be sent if a single record is between ~0.9999 MB and 1 MB. Aggregated Kinesis records have a format and contain more data than a naked record, so there is a small chance that the naked record cannot actually fit inside an aggregated record and be successfully sent to the Kinesis service (due to the 1 MB record service limit).
This record is silently dropped, but since this causes an issue identical to this, it may be worth raising the same error for this edge case.
How Has This Been Tested?
This is running on some high-volume production data pipelines that are known to have this edge case.
Types of changes
Checklist: