Update Agent to be more resilient in case of unauthenticated timeouts with IMDS

[hozkaya2000]

Summary

There is an edge case where the credentials within the agent used to access internal resources can be incorrect due to other processes restarting during a certain time interval. This can cause the agent to think that the credentials are not expired and correct. The change is to prevent this from happening, and refreshing the credentials in the first location this issue would appear, which would be RegisterContainerInstance in the ecsclient.

Implementation details

The issue occurs in the RegisterContainerInstance function in ecsclient's client.go. RegisterContainerInstance makes the call to IMDS in SetInstanceIdentity function where the client for communicating with IMDS is used with the call client.ec2metadata.GetDynamicData(...). The new IMDS process does not work with the current credentials.

In the fix, this line is replaced with a retry strategy. When we get an error code, the current credentials in agent are forced before the retry to be expired, and a GetCredentials call is made to refresh the credentials, with multiple retries if it fails again. The global instance credentials in the agent are refreshed with this call, so other calls made by agent to IMDS are no longer an issue.

Testing

To test, a new case was added to the RegisterContainerInstanceTest that is the exact same as the basic success case, but the first call to client.ec2metadata.GetDynamicData(...) was simulated to fail, and the retry call to succeed. The rest of the test validations remained, making sure that retries didn't prevent any other process from happening.

New tests cover the changes: yes

Description for the changelog

Files:

• agent/api/ecsclient/client.go
• agent/api/ecsclient/client_test.go

were changed as described in the implementation and testing details

Licensing

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[Realmonia]

Please force push to remove the internal link from commit message.