AWS sdk session init latency

[omriykl]

Please fill out the sections below to help us address your issue.

Version of AWS SDK for Go?

1.25.41

Version of Go (go version)?

1.13.3

What issue did you see?

After upgrading aws-sdk-go from version 1.25.36 to 1.25.41 we experienced high latency (~ 10 minutes) with initialization of the aws session by our app.

After ~10 minutes since starting the app we have seen in the log the following error:
“request expired, resigning”.
We found that the source of this error is: https://github.com/aws/aws-sdk-go/blob/master/aws/corehandlers/handlers.go#L81

Using back version 1.25.36 worked as expected.

We have checked and approved that the issue started in version 1.25.38.

Steps to reproduce

If you have an runnable example, please include it.

[diehlaws]

Hi @omriykl, thanks for reaching out to us. Is your code running in a Kubernetes pod, Docker container, or on a host using any IP forwarding/proxy? If so, you can use EC2's ModifyInstanceMetadataOptions call to update the hop limit needed for your application's use case. If reducing the hop limit does not reduce the time taken to retrieve instance profile credentials, please enable debug logging and CredentialsChainVerboseErrors in your session config and provide the resultant output for further review on our end.

Version 1.25.38 of the AWS SDK for Go introduced some changes to support new secure data flow for calls to the Instance Metadata Service. You can read more about EC2's IMDS update here. This should not affect sessions initialized using credentials retrieved from environment variables, the shared credentials file (~/.aws/credentials), or hard-coded into the session's config - if you are using one of these methods to initialize your session please do let us know so we can further investigate this behavior - enabling debug logging and verbose credential chain errors as mentioned above would help in this case as well.

[aaron42net]

This is in Docker on k8s using IMDS-provided roles via kiam, yes.

Wow this isn't the response I was expecting. AWS is intentionally breaking all IMDS usage in Docker containers in a minor update to the Go SDK? Really? The release notes for 1.25.38 only mention "Adds support for EC2Metadata client to use secure tokens provided by the IMDS", not any warning about breaking existing usage.

How do I disable the use of secure tokens on the Go SDK side? I'd like to upgrade the Go SDK independently of making changes to our entire fleet of EC2 instances.

[aaron42net]

This issue can be closed as a duplicate of #2972. #2980 or similar is what we're asking for.

[lifeofguenter]

terraform is also affected since provider version 2.39 - seems like a breaking change to me. I am surprised it has not affected many yet...