v2.4.0 #1751
Closed
rubenfonseca
started this conversation in
Show and tell
v2.4.0
#1751
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Summary
This release introduces a brand new streaming utility to handle datasets larger than the available memory.
This addresses scenarios where you want to process Amazon S3 objects as seekable data streams, with minimal memory consumption.
Built-in data transformations
We've added support for popular data transformations to decompress and deserialize data (gzip, CSV and ZIP). This way you can transform the data while streaming, and only use the minimum amount of memory necessary.
Bring your own data transformation
You can also create your own data transformations and add them to the pipeline. For example, here's how to use the
ijsonlibrary to parse JSON as a data stream:We plan to support more data transformations in the future too!
🌟 Would you like to see a specific data transformation supported in Powertools? Drop us a line in the python channel of our Discord.
Changes
🌟New features and non-breaking changes
📜 Documentation updates
🔧 Maintenance
This release was made possible by the following contributors:
@dependabot, @dependabot[bot], @heitorlessa, @rubenfonseca and Release bot
This discussion was created from the release v2.4.0.
Beta Was this translation helpful? Give feedback.
All reactions