Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws s3 driven adapter, generates a OOM when reading a file big enough. #374

Closed
gabheadz opened this issue Jul 4, 2023 · 2 comments
Closed
Assignees
Labels

Comments

@gabheadz
Copy link
Contributor

gabheadz commented Jul 4, 2023

Is your feature request related to a problem? Please describe.
aws s3 driven adapter, generates a OOM when reading a file big enough. Current implementation calls for returning a InputStream with AsyncResponseTransformer.toBytes which tries to "write all contents to a byte array" according to Aws SDK documentation.

Describe the solution you'd like
Provide a flag or configuration option to choose from the following 3 options available in the AsyncResponseTransformer parameter for method S3AsyncClient.getObject

  1. AsyncResponseTransformer.toPublisher
  2. AsyncResponseTransformer.toBlockingInputStream
  3. AsyncResponseTransformer.toBytes

For reactive projects options 1 or 3 may be used, with 1 as default. And for imperative projects options 2 or 3 may be used, with option 2 as default. Developers should only consider option 3 for reading small files only.

Describe alternatives you've considered
None at the time of writing this issue.

Additional context
N/A

@gabheadz gabheadz self-assigned this Jul 24, 2023
@gabheadz
Copy link
Contributor Author

Ultimately it would end having only the AsyncResponseTransformer.toPublisher, returning a Flux of ByteBuffer's for reactive projects.

@github-actions
Copy link

🎉 This issue has been resolved in version 3.6.1 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant