Support aws s3 cp from Query String Authentication URLs

[lrowe]

We control access to our files stored on S3 externally from our web application by sending a 302 redirect to a signed url, generated using boto.s3.connection.S3Connection.generate_url.

Many of our files are large (multi-GB genome sequence data) and we'd like to allow users to download these using aws s3 cp as we've found it to be significantly faster than alternative parallel download utilities such as pcurl.sh and lftp.

Ideally this would support following a 302 redirect to the signed URL, so that our users could simply run:

$ aws s3 cp "https://user:pass@example.com/redirect-to-signed-s3-url" .

[lrowe]

One complicating factor here, aws s3 cp will first perform a HEAD request to find the size of the object, using that to determine the number of range requests to use. Unfortunately a signed GET url is not valid for a HEAD request as the request method forms part of the signed data... This is a right pain and contrary to the HTTP spec which calls on servers to send the same headers for HEAD as for GET requests (https://tools.ietf.org/html/rfc7231#section-4.3.2). I'd be very happy to see S3 accept GET signatures for HEAD requests.

[ASayre]

Good Morning!

We're closing this issue here on GitHub, as part of our migration to UserVoice for feature requests involving the AWS CLI.

This will let us get the most important features to you, by making it easier to search for and show support for the features you care the most about, without diluting the conversation with bug reports.

As a quick UserVoice primer (if not already familiar): after an idea is posted, people can vote on the ideas, and the product team will be responding directly to the most popular suggestions.

We’ve imported existing feature requests from GitHub - Search for this issue there!

And don't worry, this issue will still exist on GitHub for posterity's sake. As it’s a text-only import of the original post into UserVoice, we’ll still be keeping in mind the comments and discussion that already exist here on the GitHub issue.

GitHub will remain the channel for reporting bugs.

Once again, this issue can now be found by searching for the title on: https://aws.uservoice.com/forums/598381-aws-command-line-interface

-The AWS SDKs & Tools Team

[jamesls]

Based on community feedback, we have decided to return feature requests to GitHub issues.

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than one year. We encourage you to check if this is still an issue in the latest release. Because it has been longer than one year since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment to prevent automatic closure, or if the issue is already closed, please feel free to reopen it.