request compression

[dlm6693]

This pull request enables request compression in the AWS Python SDK. It is on by default, but only used if the requestcompression trait is attached to the operation being called. At the moment, the only compression algorithm supported is gzip, but this is subject to change in the future. This also addresses a few typos I discovered while working on this project. Below is a summary of the new features in this PR.

• Add request_min_compression_size_bytes setting to Config object and config file, and support for AWS_REQUEST_MIN_COMPRESSION_SIZE_BYTES environment variable.
  • If not provided by a user, the value will default to 10240. The value MUST be a parseable integer between 0 and 1048576 (1024**2).
• Add disable_request_compression setting to Config object and config file, and support for AWS_DISABLE_REQUEST_COMPRESSION environment variable.
  • If not provided by a user, the value will default to False. If not a parseable boolean, the value will be coerced to False, similar to other config settings that are boolean types in botocore.
• Add request_compression property to OperationModel. This will allow the SDK to check for the presence of this operation trait and run compression if so.
• Add the compress.py file. This file contains logic that is actually responsible for compressing requests. It has one public API:
  • maybe_compress_request: This checks whether compression should be performed, actually does compression if so and injects the corresponding Content-Encoding header if needed.
• Compression is supported for all types of bodies the SDK already supports: str, bytes, bytearray, and file-like objects like io.StringIO and io.BytesIO for example.
• Compression is called just before flexible checksums are calculated in BaseClient._make_api_call
• Besides the above new settings, compression will also not occur if a customer has configured signature_version to be v2
• In order to support compression for services using the query protocol, dictionary bodies are serialized to bytes if the operation supports compression and the customer hasn't explicitly disabled compression.

[codecov-commenter]

Codecov Report

Patch coverage: 100.00% and project coverage change: +0.05% 🎉

Comparison is base (6bd4e24) 93.36% compared to head (494c7e3) 93.41%.
Report is 34 commits behind head on develop.

❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2959      +/-   ##
===========================================
+ Coverage    93.36%   93.41%   +0.05%     
===========================================
  Files           65       66       +1     
  Lines        13841    13948     +107     
===========================================
+ Hits         12923    13030     +107     
  Misses         918      918              

Files Changed	Coverage Δ
botocore/config.py	100.00% <ø> (ø)
botocore/args.py	100.00% <100.00%> (ø)
botocore/client.py	97.68% <100.00%> (+<0.01%)	⬆️
botocore/compress.py	100.00% <100.00%> (ø)
botocore/configprovider.py	94.78% <100.00%> (ø)
botocore/model.py	98.38% <100.00%> (+0.01%)	⬆️

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[jonemo]

I focused on how config values are handled and the AWSRequestCompressor class so far. Taking a break now and will continue with the other parts in the next review session.

[nateprewitt]

One minor comment, but once tests pass, I think this should be set.

[nateprewitt]

Looks good pending passing tests, thanks @dlm6693!