Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Introduce AWS SQS Sink #207

Merged
merged 2 commits into from
Mar 17, 2022
Merged

feat: Introduce AWS SQS Sink #207

merged 2 commits into from
Mar 17, 2022

Conversation

scarmuega
Copy link
Member

This PR introduces a new sink that sends each event to an AWS SQS message queue.

  • a new feature flag aws is added to enable the feature / dependencies
  • It supports both FIFO and non-FIFO queues
  • For FIFO queues, it uses a configurable message group id and the event fingerprint for the message id.
  • It uses the default credentials provider chain logic as common AWS libraries (env vars, .aws/credentials, EC2 context, etc)
  • the built-in retry logic of the library is used for failed requests. Max retries is configurable at the sink-level.

Known Limitations:

  • SQS has a 256kb payload size limit. This is good enough for individual events, but it won't handle blocks when the include_cbor_hex option is enabled.

@scarmuega scarmuega added the enhancement New feature or request label Mar 17, 2022
@scarmuega scarmuega added this to the v1.3 milestone Mar 17, 2022
@scarmuega scarmuega self-assigned this Mar 17, 2022
@scarmuega scarmuega marked this pull request as ready for review March 17, 2022 12:17
@scarmuega scarmuega requested a review from rvcas March 17, 2022 13:05
Copy link
Member

@rvcas rvcas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool

@scarmuega scarmuega merged commit 064c7e6 into main Mar 17, 2022
@scarmuega scarmuega deleted the feat/aws branch March 17, 2022 21:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
No open projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants