Cache temporary STS credentials

[fxaguessy]

As far as I know, contrary to awscli, there is no caching of temporary STS credentials in aws-sdk-go, for example when using MFA.
This feature was discussed in #841, but according to the MFA discussion in #842 and implementation in #1088, I don't think that this was implemented. Am I wrong ?

We are building a CLI relying on aws-sdk-go, and several users ask for this feature (cf wallix/awless#104 or wallix/awless#109). As in a CLI, the life of the process and session is very short, when using MFA, a code may be asked very often. Did I miss something or is it a new credential provider that we need to add ?

[jasdel]

@fxaguessy thanks for reaching out to us. You are correct the aws-sdk-go does not implement support for a cross-process credentials cache. The only caching of credentials the SDK includes is in process via a Session, via the Config's Credentials.

Being able to cache credentials cross process for a CLI makes a lot of sense, especially for MFA. I could imagine that being painful to use with MFA and STS. The AWS SDKs do not support the AWS CLI's file based credential cache as that cache is owned as internal functionality of the CLI. None of the SDKs use this cache for that reason.

I think the best approach to this problem would be to create a cache that is specific to the awless use case. Creating a standalone cache separate from the AWS CLI's cache would be good to avoid being impacted by any breaking change the CLI may make to its cache in the future.

I think there is some work we can do on the aws-sdk-go to make injecting a cache easier. Today adding a cache on top of the SDK, while also maintaining default credential chain functionality created by the Session, is the following:

type FileCacheProvider struct {
	Creds *credentials.Credentials
}

func (f *FileCacheProvider) Retrieve() (credentials.Value, error) {
	// TODO check file credential cache before looking at nested credentials.
	// Fall back to underlying credentials, and repopulate the cache.
	return f.Creds.Get()
}
func (f *FileCacheProvider) IsExpired() bool {
	// TODO check file cache is expired? Fall back to underlying credentials
	return f.Creds.IsExpired()
}

func main() {
	sess := session.Must(session.NewSession())

	// Inject cache able credential provider on top of the SDK's credentials loader
	sess.Config.Credentials = credentials.NewCredentials(&FileCacheProvider{
		Creds: sess.Config.Credentials,
	})

	// create service clients with sessions and make API calls.
}

The way the SDK supports the shared config makes wrapping injecting the FileCacheProvider prior to creating the Session very complicated. The above example is probably the best way to do this today after the Session is created.

This is an area the SDK can be improved I think. At the minimum if the Credentials type exposed its Provider you'd be able to easily wrap the underlying provider. Wrapping the Provider instead of the Credentials adds a significant benefit as it removes the synchronous locking overhead the Credentials type uses to ensure it is safe to use service clients across multiple goroutines.

[fxaguessy]

Many thanks @jasdel for this great answer. This confirms what I thought, but your mock will be a good start for the implementation of the cache of credentials in awless.

I'm not sure to have fully understood this:

Wrapping the Provider instead of the Credentials adds a significant benefit as it removes the synchronous locking overhead the Credentials type uses to ensure it is safe to use service clients across multiple goroutines.

But I will start the implementation and if I find problems, I will come back and discuss it here.

[jasdel]

Thanks for the update @fxaguessy, glad that was helpful.

I'm not sure to have fully understood this:

Wrapping the Provider instead of the Credentials adds a significant benefit as it removes the synchronous locking overhead the Credentials type uses to ensure it is safe to use service clients across multiple goroutines.

Sure, let me clarify that. This is an error the SDK could improve with a minor optimization by exposing the credentials Provider built by the Session instead of needing to wrap the Session's Credentials value. The Credentials type implements locking so it can be used across multiple goroutines safely. By wrapping the Session's Credentials value with another credentials Provider and wrapping that provider in a Credentials it duplicates the locking overhead by the SDK retrieving credentials, and checking if they are expired.

[jasdel]

Hi @fxaguessy I created PR #1320 a earlier this week that adds support for Go 1.8's plugin to retrieve AWS credentials from. I think there are use cases where users of awless might find it useful to provide custom credential sources for the CLI to use, without needing to modify the CLI's code. If you get a chance to look at it let us know what you think of the API and its usefulness.

The PR includes example and documentation on how to use this new feature. This feature is opt in and needs to be explicitly configured by the application using the SDK.

[fxaguessy]

Thanks @jasdel for the suggestion. This is indeed a useful feature, and we might integrate that into awless, in addition to STS credentials caching. The main limitation of Go 1.8's plugins for now is that it only works on Linux, but this might interest some users, for sure.
I will let you know if I have some feedback on this feature.

[fxaguessy]

Hi @jasdel, I started a POC implementation of the FileCacheProvider, beginning with the code you provided above (thanks again !).

It is a good start, as it successfully caches MFA credentials during multiple awless commands. However, I don't know how to check if/when the credential is expired (cf. wallix/awless@4048c5e#diff-fbe390e32c27fafd0343c9684ffd6686R161). As I get credentials from f.Creds.Get(), stored in a credentials.Value, I can not get back to the original provider with the expiration property.

Did I miss something or should I do a query to AWS, catching the RequestExpired: Request has expired error ? As a hack, I could also store time.Now().Add(stscreds.DefaultDuration) as expiration date, but it doesn't seems to be a good solution either.

[jasdel]

From a Credentials value you should be able to call IsExpired method to determine if the underlying credentials are expired or not. Your FileCacheProvider wrapping the SDK's build Credentials value will be able to do this.

Your application won't be able to know what time the credentials will expire on, it will know if they are expired or not. To handle this when credentials expire you'll need a way to lock the file cache so only a single cli instance will attempt to refresh the credentials from AWS.

[jasdel]

Oh i see part of the problem is determining if the cached credentials are expired correct?

[fxaguessy]

Yes exactly, the problem is for credentials that have been cached.

[jasdel]

hmm so I don't think the SDK's Credentials or Provider expose the value that is needed here. I think to support this the SDK's Credentials type needs to be updated so Provider is an exported field. If Provider can be retrieved from credentials this would provide the chance at getting an expiry time from the underlying provider.

We'd probably want to create a new interface Expirer in the credentials package that a Provider could satisfy if it has the ability to state when its credentials will expire.

type Expirer interface {
    ExpiresAt() time.Time
}

In this case the stscreds Provider would be updated to satisfy the Expirer interface.

[fxaguessy]

Yes, indeed, such an Expirer interface would allow to know the expiration date, before caching the credential. That would be a great help for my use case.

[posilva]

• I wrote this small library that uses some sort of in memory "cache" of MFA: https://github.com/posilva/go-mfa/
• I will extended it to cache locally the credentials to be easy to reuse without having to enter the MFA while the session token is not expired.

PR, comments and issues are welcome

[llamahunter]

The aws-iam-authenticator seems like it could use an implementation of a disk based credentials cache. I'm unclear why they can't use ~/.aws/credentials, tho. Or why this isn't built in to the aws-sdk-go package.

[llamahunter]

See PR #2375 for an implementation of the above discussed Expirer interface on Providers. I've implemented a persistent credential cache in the aws-iam-authenticator project built on aws-sdk-go using this functionality.

[llamahunter]

See kubernetes-sigs/aws-iam-authenticator#193 for an example of writing an application level credential cache using aws-sdk-go.