Upcoming Compression changes in Boto3

[nateprewitt]

Hi there,

I'm reaching out from the Boto3 team about a recently released feature (boto/botocore#2959) that may have some implications on how moto works with request body parsing. In order to help optimize bytes sent over the wire, boto3 will slowly start onboarding some services and operations to compress their request payload by default (only gzip is supported currently). This should improve latency for particularly large calls with the end goal of streamlining throughput and performance for customers.

With this onboarding, we'll be updating our service models by adding new traits per operation where we notice particularly large request sizes. Doing some initial testing with moto, it looks like the class setup in moto.core.response.py is making some assumptions about the body that may cause issues as this is rolled out. Namely that any bytes body is assumed to be a utf-8 string which won't be true for a gzip file.

Possible Solution(s)

Decompress bodies before processing

For our implementation we utilize the standard method of Content-Encoding headers (RFC 9110 8.4) in our requests to denote a compressed body. This will likely be the most effective method to recognize these changes to request payloads in moto and decompress before doing your body processing workflow.

This should be safe to implement uniformly across all AWS services except s3 which has some specific implications around Content-Encoding in operations like PutObject where the body should be left untouched. For whatever solution is implemented, we'd recommend exempting S3 from this as they're considered out of scope for this feature.

Disable Request Compression in moto mock

I'm not immediately sure about the feasibility on this option, but we do provide an escape hatch in boto3 to explicitly disable this. Both an environment variable (AWS_DISABLE_REQUEST_COMPRESSION) and a Config option (disable_request_compression) can be used to shut off this functionality. It may be possible for moto to enable this when the mock is used to ensure the body will never be compressed.

Next Steps

We don't have firm dates on when we'll start rolling this out to services, but I can share the we're intending to start with some operations in CloudWatch. We wanted to proactively reach out to hopefully avoid the friction moto users had with the recent SQS changes that required (#6331).

I'm happy to talk through any implementation specifics or provide more information that might be needed to align on a fix in moto. I couldn't find anything generalized in the moto codebase for handling this currently so I'm assuming it will require some net-new code in the BaseResponse class unless there's a better place to handle this.

[bblommers]

Hi @nateprewitt, thanks for reaching out! This would indeed require some non-trivial changes, so this heads-up is very welcome.

Based on your description, and the PR, I think we have everything we need to start implementing support for this feature - the implementation in boto seems quite standard/straightforward.

Having an escape hatch is useful, although that's not something Moto can rely on - because users can also run Moto in ServerMode, we don't always control the invocation environment. Users would have to set the environment variable themselves, if Moto's implementation is not ready for whatever reason.

Just out of curiosity - do you have any insight on what the other SDK's will do? I assume they will follow suit and also support encoding, but I just checked, and didn't see any relevant issues/PR's in the DotNet or JS repositories.

[nateprewitt]

I can't speak to specific timelines on the other SDKs, but they should be eventually consistent across each languages latest major version.