AWS Data Wrangler 2.3.0

New Functionalities

• DynamoDB support #448
• SQLServer support (Driver must be installed separately) #356
• Excel files support #419 #509
• Amazon S3 Access Point support #393
• Amazon Chime initial support #494
• Write compressed CSV and JSON files on S3 #308 #359 #412

Enhancements

• Add query parameters for Athena #432
• Add metadata caching for Athena #461
• Add suffix filters for s3.read_parquet_table() #495

Bug Fix

• Fix keep_files behavior for failed Redshift COPY executions #505

Thanks

We thank the following contributors/users for their work on this release:

@maxispeicher, @danielwo, @jiteshsoni, @gvermillion, @rodalarcon, @imanebosch, @dwbelliston, @tochandrashekhar, @kylepierce, @njdanielsen, @jasadams, @gtossou, @JasonSanchez, @kokes, @hanan-vian @igorborgest.

---

P.S. The AWS Lambda Layer file (.zip) and the AWS Glue file (.whl) are available below. Just upload it and run!

AWS Data Wrangler 2.2.0

New Functionalities

• Add aws_access_key_id, aws_secret_access_key, aws_session_token and boto3_session for Redshift copy/unload #484

Bug Fix

• Remove dtype print statement #487

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @thetimbecker, @njdanielsen, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 2.1.0

New Functionalities

• Add secretmanager module and support for databases connections #402

con = wr.redshift.connect(secret_id="my-secret", dbname="my-db")
df = wr.redshift.read_sql_query("SELECT ...", con=con)
con.close()

Bug Fix

• Fix connection attributes quoting for wr.*.connect() #481
• Fix parquet table append for nested struct columns #480

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @nmduarteus, @nivf33, @kinghuang, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 2.0.1

New Functionalities

• New wr.timestream.create_database() function
• New wr.timestream.create_table() function
• New wr.timestream.delete_database() function
• New wr.timestream.delete_table() function
• New ignore_empty argument to ignore 0 bytes files for:
  • wr.s3.merge_datasets()
  • wr.s3.list_objects()
  • wr.s3.read_parquet()
  • wr.s3.read_parquet_metadata()
  • wr.s3.read_csv()
  • wr.s3.read_fwf()
  • wr.s3.read_json()
  • wr.s3.store_parquet_metadata()

Enhancements

• Automatically rollback in case of failed queries for:
  • wr.redshift.read_sql_query()
  • wr.postgresql.read_sql_query()
  • wr.mysql.read_sql_query()

Thanks

We thank the following contributors/users for their work on this release:

@danielwo, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 2.0.0

Breaking changes

• sqlalchemy and psycopg2 dependencies replaced by redshift_connector and pg8000
• All wr.db.* functions was distributed into wr.redshift.*, wr.postgresql.* and wr.mysql.* (Tutorial)
• Redshift COPY and UNLOAD function was refactored into wr.redshift.* (Tutorial)
• wr.catalog.get_engine() was replaced by wr.redshift.connect(), wr.postgresql.connect(), wr.mysql.connect() (Tutorial)

New Functionalities

• Amazon Timestream support (Tutorial)

Enhancements

• General performance improved for s3 I/O removing eventual consistency guardrails (Reference)
• Add retry with decorrelated jitter for Athena and Glue Catalog calls to overcome throttling in high concurrency scenarios.

Docs

• Updates regarding all new functionalities
• Add Amazon Timestream tutorial
• Add Amazon Timestream tutorial 2

AWS re:Invent related news

• AWS Lambda now supports up to 10 GB of memory and 6 vCPU cores
• Amazon S3 now delivers strong read-after-write consistency
• AWS Lambda now supports container images as a packaging format
• Serverless Batch Scheduling with AWS Batch and AWS Fargate

Thanks

We thank the following contributors/users for their work on this release:

@Brooke-white, @danielwo, @sapientderek, @pmleveque, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 1.10.1

New Functionalities

• catalog.add_column() #451
• catalog.delete_column() #451

Enhancements

• Deterministic result for s3.read_parquet_metadata() #449
• ~30% faster package import time #460

Bug Fix

• Fix Athena read with ctas_approach=False and chunksize=True #458
• Fix overwriting for not enforced configs #450

Docs

• Small fixes #462 #458 #446

Thanks

We thank the following contributors/users for their work on this release:

@tuannguyen0901, @bryanyang0528, @czagoni, @jesusch, @danielwo, @DonghanYang, @eric-valente, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 1.10.0

New Functionalities

• Add configurable Endpoint URL for AWS services #418
• Add global environment configuration for Athena workgroups #437

Enhancements

• Support for Apache Arrow 2.0.0 #436
• Allow Decimal to float casting for wr.db.read_sql_query() #431
• Allow unsafe conversions for wr.db.read_sql_query() #427

Bug Fix

• QuickSight functions now allow usernames with "/" #434
• Fix duplicated carriage return for wr.s3.to_csv() running on Windows platform.

Thanks

We thank the following contributors/users for their work on this release:

@martinSpears-ECS, @imanebosch, @Eric-He-98, @brombach, @Thomas-Hirsch, @vuchetichbalint, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 1.9.6

Enhancements

• Add encrypted glue connection management #413

Bug Fix

• Double carriage return when using \r\n as line terminator (s3.to_csv()) #415
• s3.read_parquet failing with some timezone aware columns #417

Thanks

We thank the following contributors/users for their work on this release:

@jeanbaptistepriez, @mike-at-upside, @Thiago-Dantas, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 1.9.5

Enhancements

• General exceptions handling improvements #409
• General error messages improvements #409

Bug Fix

• [Parquet Read] Fix index recovery combined with columns filter #408

Docs

• Handling and documenting ctas_approach for custom data sources #392

Thanks

We thank the following contributors/users for their work on this release:

@tasq-inc, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!

AWS Data Wrangler 1.9.4

Enhancements

• Add s3_additional_kwargs for wr.s3.copy_objects() and wr.s3.merge_datasets() #388
• Add data_source argument for Athena queries #392
• Handling parquet tinyint columns on Redshift loads #400

Bug Fix

• Fix issue with Hive partitions compatibility. #397
• Fix missing catalog_id arguments in partitioned wr.s3.to_parquet() calls #399
• Remove adaptive retry for boto3 resource. #403

Docs

• Few updates.

Thanks

We thank the following contributors/users for their work on this release:

@timgates42, @bvsubhash, @DonghanYang, @sl-antoinelaborde, @Xiangyu-C, @tuannguyen0901, @JPFrancoia, @sapientderek, @igorborgest.

---

P.S. Lambda Layer zip file and Glue wheel/egg files are available below. Just upload it and run!