s3 client signature failure - with known good credentials

[dcopenhaver]

Describe the bug

operation error S3: PutObject, https response error StatusCode: 403, RequestID: REDACTED, HostID: REDACTED, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method

Expected Behavior

successful putObject request

Current Behavior

operation error S3: PutObject, https response error StatusCode: 403, RequestID: REDACTED, HostID: REDACTED, api error SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your key and signing method

Reproduction Steps

// the httpClient used simply changes the default timeout

ctx := context.Background()

cfg, err := config.LoadDefaultConfig(
ctx,
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKeyId, secretAccessKey, "")),
config.WithHTTPClient(httpClient),
config.WithRegion("us-east-1"),
)

if err != nil {
fmt.Printf("Error loading AWS SDK config.\n%v", err)
return
}

// create s3 service client
s3Client := s3.NewFromConfig(cfg)

// upload object to bucket
_, err = s3Client.PutObject(ctx, &s3.PutObjectInput{
Bucket: aws.String("theBucketName"),
Key: aws.String("theKeyName"),
Body: strings.NewReader("This is a test value"),
})

if err != nil {
fmt.Printf("ERROR occured during PutObject request:\n%v", err)
os.Exit(1)
}

Possible Solution

No response

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

github.com/aws/aws-sdk-go-v2 v1.17.7
github.com/aws/aws-sdk-go-v2/config v1.18.19
github.com/aws/aws-sdk-go-v2/credentials v1.13.18
github.com/aws/aws-sdk-go-v2/service/s3 v1.31.0

Compiler and Version used

go1.19.4 windows/amd64

Operating System and version

Windows 11 Pro (10.0.22621 N/A Build 22621)

[RanVaknin]

Hi @dcopenhaver ,

Your code looks ok, I don't see any specific reason why you might run into a 403. Are you running this through a proxy, or from a context of a lambda or a container that might overwrite some headers?
Can you please share your implementation for httpClient?
Additionally enabling request and response logs might give us some more useful info. You can enable logs by doing the following:

		config.WithClientLogMode(aws.LogRequestWithBody|aws.LogResponseWithBody),

Make sure you redact any sensitive information before sharing your logs.

Thanks,
Ran~

[dcopenhaver]

Hi @RanVaknin,

Thanks for the quick response. No proxy, direct connection from my dev machine to aws. Here is the httpClient:

// Create an HTTP client with a custom timeout
httpClient := &http.Client{
    Timeout: 15 * time.Second,
}

I'll get the debug info shortly...

• David C.

[dcopenhaver]

Hi @RanVaknin,

Here's the extra logging info. I errored on the side of caution removing sensitive or potentially sensitive info:

SDK 2023/03/27 14:01:06 DEBUG Request
PUT /ncslot1?x-id=PutObject HTTP/1.1
Host: REDACTED.s3.us-east-1.amazonaws.com
User-Agent: aws-sdk-go-v2/1.17.7 os/windows lang/go/1.19.4 md/GOOS/windows md/GOARCH/amd64 api/s3/1.31.0
Content-Length: 20
Accept-Encoding: identity
Amz-Sdk-Invocation-Id: REDACTED
Amz-Sdk-Request: attempt=1; max=3
Authorization: AWS4-HMAC-SHA256 Credential=REDACTED/20230327/us-east-1/s3/aws4_request, SignedHeaders=accept-encoding;amz-sdk-invocation-id;amz-sdk-request;content-length;content-type;host;x-amz-content-sha256;x-amz-date, Signature=REDACTED
Content-Type: application/octet-stream
X-Amz-Content-Sha256: UNSIGNED-PAYLOAD
X-Amz-Date: 20230327T180106Z

This is a test value
SDK 2023/03/27 14:01:06 DEBUG Response
HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Mon, 27 Mar 2023 18:01:07 GMT
Server: AmazonS3
X-Amz-Id-2: REDACTED
X-Amz-Request-Id: REDACTED

I didn't include the response data that came after as I'm assuming what's already reported here is enough for that side, 403, and 'SignatureDoesNotMatch'. Let me know if there is something in there I should include though and I'll grab it.

Thanks and Regards,

• David C.

[RanVaknin]

Hi @dcopenhaver ,

This is indeed odd. Your logs look good, your code looks correct, I'm able to run it successfully on my machine.
Since the SDK is a client and does not maintain logs, we will need to investigate this upstream with the S3 service team. At this point I would suggest you grab that x-amz-request-Id value you have there in the response log and reach out to AWS support.

With AWS support you will be assigned a person that will be able to reach out to any team in AWS.
The S3 service team will hopefully be able to locate the request made, investigate it and tell us why the signature doesn't match.

If you don't have access to support / can't do it, please let me know and I can file the internal ticket on your behalf.

Thanks,
Ran~

[dcopenhaver]

Hi @RanVaknin,

Ok, understood. I'm going to completely remove the involved components from my system and start a fresh module/project and retry. I'm thinking I should do this as I did do a lot 'go get', 'go get -u', on the original version of aws-sdk-go and then changing to aws-sdk-go-v2 after realizing that was available. So maybe I just made a mess of my local system during that process (though I did be sure to 'go mod tidy'). If it still fails I'll create the AWS ticket. Either way I'll update back here.

Thanks for checking into this, much appreciated.

• David C.

[RanVaknin]

Hi @dcopenhaver ,

I think starting fresh is a good idea, however from the useragent string Im able to tell that you are using the correct version of the SDK. The only thing I can think of is that you are using some special characters in your bucket name that are not getting serialized correctly and causing a signatureMismatch error? At this point its all speculation.

Please let me know if you are able to fix the issue or if you need help with anything else.
Thanks,
Ran~

[dcopenhaver]

@RanVaknin,

The issue is resolved. It turned out to be something very unexpected and very much nothing to do with aws-sdk-go-v2. I was retrieving my aws credentials from the windows credential store which was storing the secretAccessKey part as UTF16 encoded bytes. So even though when I would fmt.Println(secretAccessKey) to verify my variable was holding the correct value it LOOKED right due to secretAccessKey=string(credBlob), whereas string() is smart enough, or the console also, to 'display' the string correctly but when I checked the output of secretAccessKey=="the hard coded secret" it came back FALSE! Converting the UTF16 to UTF8 was the fix for me.

Thanks again for looking into this. I don't think I would have discovered that without knowing everything else looked fine and worked for you.

Regards,

• David C.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.