Use AWS.Response to handle streaming/raw/parsing

[omus]

The idea is to use a struct in AWS.jl that can be used to handle the automatic parsing that is currently used to turn XML/JSON into a dict while also giving the option of accessing the raw output as a string or stream without all the keywords currently needed to be specified.

Depends on:

• Support specifying features with @service #457

Related:

• RFC: Return Raw more explicitly #346
• AWS.jl@2 #348
• consider switching the JSON3 #438

Closes:

• S3: force get_object to return Vector{UInt8}? #224
• [Downloads backend] are we closeing streams correctly? #433
• Should _http_request only close streams it has created? #468 (when using use_response_type the passed in I/O is not closed)
• DownloadsBackend inconsistently closes streams #471

Update: The tests in this PR run using the deprecated behaviour. Mainly I did that here to prove this change is non-breaking. For the updated tests see #458

[omus]

At the moment I've made the basic AWS.Response structure and I am using it as part of submit_request. I've also introduced a shim into the service request calls to re-introduce the legacy behaviour while I'm working through the details

[omus]

bors try

[bors]

try

Build failed:

• Julia 1 - ubuntu-latest - x64

[omus]

I'll pick this up again next week. Lots of details to work though yet but I think overall this is looking like the right approach. There is still a question of how to make this very user friendly for newcomers

[omus]

bors try

[bors]

try

Build failed:

• Julia 1 - macOS-latest - x64

[omus]

Things feel like they are starting to align here. Overall the concept of returning a AWS.Response type seems to be the right approach: it allows us to use dispatch to process the response in several different ways and allows end users to process returned data with whatever tooling they want (#438).

The change is less breaking than I originally expected as the AWS.Response implements the dictionary interface which provides some backwards compatibility. I need to think through the various other ways things were previously being handled yet to make sure I'm not missing something though.

I'm sure I've got some rounds of debugging to do yet but the only major change I anticipate is I still may want to support passing in a user supplied IO type. The reason for this is that it allows users to write directly to disk or to some other source where as currently we're always using an IOBuffer. I also need to validate the alternate download backends are working correctly.

Update: User supplied IO instance is now supported

[omus]

bors try

[bors]

try

Build failed:

• Julia 1 - ubuntu-latest - x64

[omus]

bors try

[bors]

try

Build failed:

• Julia 1 - ubuntu-latest - x64

[omus]

bors try

[bors]

try

Build failed:

• Julia 1 - ubuntu-latest - x64

[omus]

bors try

[bors]

try

Build failed:

• Julia 1.3 - ubuntu-latest - x64

[omus]

Validated that #458 is working this these rebased changes: #458 (comment)

[omus]

bors r+

[bors]

Build succeeded:

• Julia 1.3 - ubuntu-latest - x64
• Julia 1.5 - ubuntu-latest - x64
• Julia 1 - macOS-latest - x64
• Julia 1 - ubuntu-latest - x64