How to generate S3 put object URL with metadata in the query string instead of headers

[alexbilbie]

Version of AWS SDK for Go?

v1.10.24-1-g141625ad

Version of Go (go version)?

go1.8.3 darwin/amd64

What issue did you see?

When I use the AWS PHP SDK to generate an S3 Presigned PUT object request the metadata map is included in the query string of the generated pre-signed URL. This means that the client doesn't have to send the values of the metadata map as x-amz-meta-* headers:

https://s3-eu-west-1.amazonaws.com/my-bucket-name/some-object-key?x-amz-meta-user-id=12345&&X-Amz-Content-Sha256=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&X-Amz-Security-Token=FQoDYXdzEBUaDHye0wAN5AeYx7KS%2FCK3Aw79D2ufXc9DnTU4B%2FDboxEED3PQfAkdn6O2PgK3QCMoQ69c1It07O1%2BMJWoZzPPCmFvU6NHPvSm76efBN3ItWFhL21VepDbp%2F9IO5dVUOXki%2FA9g08a5jwWrLMOQlsSf%2FIu3dsYdVqNQpwwmPyd3AqahvF4mC%2FGnWhMQBwn%2B7b1lk1nWlqUETFyRwLmrTK3cwhKvi6SnjbmsAZnisMr6uy0rPpr%2BzbRHhE1NBJw9gUfJ67AoiO1h1%2FjuRCiManQkd%2FqerEAppeb33fyEZhmRep7udg%2BiEQbdaYPvfV%2FJsZTusy2PzUfULX12kZg7ub%2Fvmu7%2FIQSXX8QoQ939%2FuGzocgURqEXvN59KNNMvBEqw5WPDLwpqKGnXQ5uX%2Bn9qtAEc4ATomETDu5HKy%2BnFuu%2F2isnswC1gkKhtvZLtYJguyyQbvftyXPNHSuz3nHXjE%2FJx5J8VRroLh%2F7%2BKWFuMdS0%2BYhIFnhkongCTeH5sdvoK4FPSQX%2FnMjzUFweGbZ5iFbZly1J4bwCDb0WzM4HU22TuUXn%2BMffxWdSict2I5LEw34ig1AyLS6kRszAz5ieyjW0okUtUaJRMoyZbGzAU%3D&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ASIAJYBE4F27BHJUETMA%2F20170814%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20170814T114959Z&X-Amz-SignedHeaders=Host&X-Amz-Expires=3600&X-Amz-Signature=6277eb62f8f0b0da05117e85d599349ea2f67204228044d4dd15ee08ae584663

The same code with the Go SDK generates a pre-signed URL that requires the x-amz-meta-* headers.

https://my-bucket-name.s3-eu-west-1.amazonaws.com/some-object-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJVIY4ICMYSHWGGSQ%2F20170814%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20170814T115630Z&X-Amz-Expires=10800&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256%3Bx-amz-meta-user-id&X-Amz-Signature=4edbccd736aefe762ewc29325fcb7fb6565ab1a54e876231e87fd58d66b9ed

This means I can't enforce the values of the metadata map.

How can I generate a URL in the same way with the Golang SDK?

Steps to reproduce

PHP SDK example:

$cmd = $this->s3Client->getCommand(
    'PutObject',
    [
        'Bucket'   => 'my-bucket-name',
        'Key'      => 'some-object-key',
        'Metadata' => [
            'user-id' => '12345',
        ],
    ]
);

$url = $this->s3Client->createPresignedRequest($cmd, '+1 hour');

Go SDK example:

req, _ := svc.PutObjectRequest(&s3.PutObjectInput{
    Bucket: aws.String("my-bucket-name"),
    Key:    aws.String("some-object-key"),
    Metadata: map[string]*string{
        "user-id": aws.String("12345"),
    },
})

url, headers, err := req.PresignRequest(time.Hour * 1)

[jasdel]

Hi @alexbilbie thanks for letting us know about this issue. It looks like the metadata is being added to the URL, but it is also being included in the signature of the request. This should be investigated and see if this is a bug in the signer for presigned requests.

A workaround in the short term is that the down stream client that will use the presigned URL can add the "x-amz-meta-user-id: 12345" header in addition to the header "x-amz-content-sha256: UNSIGNED-PAYLOAD" and that is already being added by the down stream client. This workaround still allows you to specify additional data, but at the cost of the down stream client specifying the fields also. The field specified by the client must match the value in the header.

[alexbilbie]

I've taken a look at another implementation we have written using the JavaScript SDK and that too embeds everything in the URL and doesn't require the client to send headers

[jasdel]

@alexbilbie after investigating this issue a bit the issue is caused by using Presign vs PresignRequest. In your case I think using Presign would accomplish what you're looking for.

PresignRequest will return a presigned URL, but all of the requests headers will be signed and not added to the URL. Looks like the docs for this method need to be improved.

[alexbilbie]

@jasdel unfortunately Presign doesn't work earlier.

Here's the URL provided by PresignRequest:

https://my-bucket-name.s3-eu-west-1.amazonaws.com/some-object-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJVIY4ICMYSHWGGSQ%2F20170814%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20170814T200914Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host%3Bx-amz-content-sha256%3Bx-amz-meta-user-id&X-Amz-Signature=8ab49e7bbc45f944cf5bc58898d813dd3e1aa81ccfd6be2f6c117dac600e061f

and the accompanying required headers:

x-amz-meta-user-id: 12345
x-amz-content-sha256: UNSIGNED-PAYLOAD

Here's the URL provided by Presign:

https://my-bucket-name.s3-eu-west-1.amazonaws.com/some-object-key?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAJVIY4ICMYSHWGGSQ%2F20170814%2Feu-west-1%2Fs3%2Faws4_request&X-Amz-Date=20170814T201106Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host%3Bx-amz-meta-user-id&X-Amz-Signature=d85a5cdc560b690fc88dae6e030ba4578f7ec7a1ccdbb3dc0d4dd8c96bc636d3

In this case you still need to send the headers for the request to validate.

[jasdel]

Thanks for the update. For Presign method of generating the URL are you seeing signature errors response from s3?

In investigating this we did discover an issue with sending the metadata over as query parameter with title casing causes their values to not be set on the S3 object's metadata. e.g X-Amz-Meta-User-Id vs x-amz-meta-user-id.

[alexbilbie]

Sending data to the Presign URL then it does produce a signature error unless the x-amz-meta-user-id header is set

[jasdel]

Thanks for the update I'm investigating the x-amz-meta- issue. But for not this seems to be a bug with S3 not handling metadata values hoisted to the query strings for presigned URLs correctly. I created #1469 which will resolve the need for the downstream client to add x-amz-content-sha256 in its usage of the presigned URLs.

[jasdel]

Hi @alexbilbie I've submitted a change in #1469 that removes the need of setting X-Amz-Content-Sha256 header in a presigned URL.

This does not solve the problem of X-Amz-Meta-* headers needing to be included in headers though. Specifically because of the issue with S3 not accepting metadata fields correctly via the query string. Only via headers can metadata data be reliably set without modifying casing of the key in the query string.

I'll reach out to S3 about the metadata query string issue letting them know. I don't think the SDK should be modified if the service can correct this issue.

[jasdel]

Removed the bug tag as #1469 corrects the issue with the content sha. Metadata should be included as a header for now, until S3 is able to correct the query string issue.

[alexbilbie]

Thanks for fixing the content sha issue.

I don't suppose you could please explain what the PHP and JS SDKs are doing differently and why you consider this an S3 service issue?

Thank you!

[jasdel]

The AWS SDKs for PHP and JS look to be sending the metadata name as lower cased as the query string key. This is how I discovered the issue where S3 does not correctly handle mixed case metadata name query string keys.

In addition, while also investigating this issue I discovered that if unicode characters are used for metadata values the value stored on the S3 object is the URI escaped value not the unescaped value the user intended to send. I'm reaching out to S3 to see how these issues can be resolved.

[jasdel]

It looks like S3 does handle non-lowercase querystring keys. This is one thing preventing to SDK from sending metadata tags via the querystring. In addition we discovered that UTF-8 metadata values are represented differently in S3 if they are sent via Header vs querystring. We recommend only sending metadata via the Header which is what the SDK currently does.

Because of these items I don't think the SDK can move the metadata from the Header to querystring, as S3's handling those would institute a breaking change in functionality.

Let us know if you have any feedback, or additional questions about this issue.

[alexbilbie]

Thank you for the update @jasdel

[kennedyjustin]

I think it would be great if the SDK could expose option (disabled by default) on the v4 Signer that allows metadata to be added to the query string (lowercase, in the form of: x-amx-meta-<key>=<value>). S3 already exposes this functionality, and there has to be an easier way to allow Go SDK users to exercise it without creating a custom Sign handler (which, I believe, is the cleanest way to do so right now besides forking the entire SDK). Is there any way we can re-open this issue to support adding this option?

[kennedyjustin]

Looks like I can actually get around this by doing the following:

req, _ := s3Client.PutObjectRequest(&s3.PutObjectInput{
    Bucket:      aws.String("bucket"),
    Key:         aws.String("key"),
})

// See https://github.com/aws/aws-sdk-go/issues/1467 for why we have to do this
query := req.HTTPRequest.URL.Query()
query.Add("x-amz-meta-key", "value")
req.HTTPRequest.URL.RawQuery = query.Encode()

// Can be used in the CLI like:
//   `curl -X PUT -T $FILE_PATH $PRESIGNED_URL`
presignedUrl, err := req.Presign(PresignDurationMinutes * time.Minute)
if err != nil {
    return err
}

return nil

[Harital]

This is actually a workaround. IMHO, the presigned url´s are useful as the clients can be agnostic of where the file is stored and how it is stored. If the client of the presigned url needs to include some random header that he doesn´t know, about, it doesn´t make any sense to include the metadata into the presigned url.
Isn´t there a definitive fix?