Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(transform): Change Kinesis Payload Limit to MB #206

Merged
merged 1 commit into from
Jul 17, 2024

Conversation

jshlbrd
Copy link
Collaborator

@jshlbrd jshlbrd commented Jul 17, 2024

Description

  • Changes the payload size limit for Kinesis Data Streams from MiB to MB

Motivation and Context

Discovered an edge case where the system generated a Kinesis Data Stream payload of more than 5 MB when using the PutRecords API. Changing the limit of each record from 1 MiB (1048576 bytes) to 1 MB (1000000 bytes) fixed the problem. According to the AWS docs, Firehose uses 1 MiB as its message size limit.

How Has This Been Tested?

Tested E2E in the production data pipeline the bug was discovered in.

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.

@jshlbrd jshlbrd marked this pull request as ready for review July 17, 2024 14:26
@jshlbrd jshlbrd requested a review from a team as a code owner July 17, 2024 14:26
@jshlbrd jshlbrd merged commit 16baaa0 into main Jul 17, 2024
4 checks passed
@jshlbrd jshlbrd deleted the jshlbrd/fix/kinesis-payload-size branch July 17, 2024 14:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants