S3 message spillover

[alukach]

As described in #37 (comment), I recommend that we try a custom middleware to automatically push all responses greater than 6MBs to AWS S3 and utilize a presigned URL + 303 redirect to instruct clients to retrieve the results from that location, thereby side-stepping the AWS Lambda Response limit. A 303 (rather than a 302 or 307) is recommended as it informs clients that a GET request should be used to follow the redirect, even when the original request was a POST (docs):

This response code is often sent back as a result of PUT or POST methods so the client may retrieve a confirmation, or view a representation of a real-world object (see HTTP range-14). The method to retrieve the redirected resource is always GET.

The S3 bucket should auto-flush these messages after 1 day.

[zacdezgeo]

@alukach, the approach with the s3 redirect is pretty seamless (once we figured out the encoding error). The user experience stays exactly the same, and performance remains relatively constant from my quick and dirty benchmarking:

Time for 0 fields: 1.8640 seconds
Time for 1 fields: 1.2667 seconds
Time for 2 fields: 1.3008 seconds
Time for 3 fields: 1.4476 seconds
[...]
Time for 35 fields: 2.7537 seconds
Time for 36 fields: 2.8424 seconds
Time for 37 fields: 2.9549 seconds
Time for 38 fields: 5.3728 seconds

and now we can return as much data a user requests!

Feel free to re-request my review once the patch is done on the middleware.

[zacdezgeo]

So awesome! Thanks @alukach.

[zacdezgeo]

Will try deploying, and I'll merge if everything goes well.

[zacdezgeo]

Deployment succeeded. I just picked up on the bucket creation in cdk/aws_stack.py. Are we good to merge @alukach ?