Cache aws credentials #193

llamahunter · 2019-01-11T01:28:47Z

Fixes #183

Possibly fixes or addresses the following in an alternate way: #151, #99, #63, #140

cmd/aws-iam-authenticator/token.go

llamahunter · 2019-01-11T01:31:55Z

pkg/token/token.go

+			profile = session.DefaultSharedConfigProfile
+		}
+		// create a cacheing Provider wrapper around the Credentials
+		if cacheProvider, err := NewFileCacheProvider(clusterID, profile, roleARN, sess.Config.Credentials); err == nil {


If caching is enabled, wrap the existing Credentials (and its contained Provider) in a new caching provider.

pkg/token/filecache.go

llamahunter · 2019-01-11T01:59:49Z

/assign mattlandis

llamahunter · 2019-01-11T18:15:48Z

Hmm... I think I might be using confusing terminology. This is really caching the AWS credentials, not the kubernetes access token. Would you like me to fix up the identifiers and comments to be clearer about what's going on? Also, probably should reference aws/aws-sdk-go#1329 somewhere in the code, so that people understand why this needs to be done, as the aws-sdk-go people say it's not their responsibility to cache credentials, and any app credential caching should not share a cache with the awscli for compatibility reasons.

llamahunter · 2019-01-19T00:18:04Z

@nckturner @mattmoyer any thoughts on this? Any rework needed? Can it be merged & released?

nckturner · 2019-01-22T17:38:56Z

/assign

llamahunter · 2019-02-03T20:54:11Z

Still waiting for feedback on this. Does it need rework? Can it be merged?

llamahunter · 2019-02-09T01:33:37Z

Hey... it's been about a month now.... please comment.

CharlieC3 · 2019-02-13T21:35:32Z

Any activity on this review process? Would love to see this merged.

llamahunter · 2019-02-13T21:36:24Z

I can only ask so many times.... @CharlieC3 do you have any feedback on it?

CharlieC3 · 2019-02-13T21:48:53Z

I'm no official reviewer of this repo, but...

I'm assuming you were able to build/test this on your end @llamahunter, but I was not due to dependency issues. Given, I only tried for maybe 15 minutes :) I'll give it another go this week to see if I can test it on my end, otherwise the source code looks good at a quick glance, and the PR comments you left are helpful.

llamahunter · 2019-02-13T22:04:56Z

Hmm... I would hope the branch would be able to build. Go is pretty good about packaging all the dependent foo together. The Travis CI builds succeeded, so I think build problems are probably a local config issue on your side.

It seems to work pretty well for me. I have a number of different IAM roles I can adopt to switch to different permissions levels within the cluster. Here's a section of my .kube/config file

users:
- name: arn:aws:eks:us-east-1:########:cluster/ekstest-admin
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - ekstest
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: default
- name: arn:aws:eks:us-east-1:########:cluster/ekstest-dev
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - ekstest
      - --cache
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: dev
- name: arn:aws:eks:us-east-1:########:cluster/ekstest-ops
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - ekstest
      - --cache
      command: aws-iam-authenticator
      env:
      - name: AWS_PROFILE
        value: ops

and my corresponding .aws/config

[default]
region = us-east-1

[dev]
region = us-east-1
credential_process = okta-credential_process arn:aws:iam::########:role/aws-dev

[ops]
region = us-east-1
credential_process = okta-credential_process arn:aws:iam::########:role/aws-ops

My default profile has a corresponding entry in .aws/credentials

[default]
aws_access_key_id = XXXXXXXX
aws_secret_access_key = XXXXXXXX

and so I don't have the --cached argument for that user in my .kube/config. The other two users use okta-credential-process to get SSO credentials

pkg/token/filecache.go

nckturner

We do need something similar to this PR to relieve the pain of MFA caching. Apologies on the time to review.

pkg/token/filecache.go

cmd/aws-iam-authenticator/token.go

pkg/token/filecache.go

llamahunter · 2019-03-07T02:09:30Z

Thx for the feedback. Will review above recommendations and push new code to the PR in about a week. Currently out of the country traveling w/o computer.

Create a disk cache for credentials, based on cluster, profile, and roleARN, and use cached credentials until they expire. Requires that the credential supports Expirer interface (i.e. works for SSO credenital_process credentials, but not for static unexpiring credentials). Caching based on profile/roleARN permits easy switching of kubectl config between different user contexts by configuring different profiles using AWS_PROFILE env variable in the exec block of .kube/config

llamahunter · 2019-03-19T02:44:26Z

Addressed the review comments and rebased against master.

nckturner · 2019-04-05T15:16:42Z

Appreciate the changes, taking a look.

nckturner · 2019-04-05T18:38:50Z

Tested this PR after the changes and it works great. Let's get it merged! Specifically tested:

MFA caching
Switching between two roles with MFA
Expired credentials

/lgtm

nckturner · 2019-04-05T18:56:29Z

/approve

k8s-ci-robot · 2019-04-05T18:56:36Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: llamahunter, nckturner

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [nckturner]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

robwittman · 2019-04-08T18:35:38Z

🙌 Any timeline as to when we can see this in a release?

nckturner · 2019-04-09T18:54:09Z

@robwittman working on it! Hopefully this week.

joegoggins · 2019-05-02T17:38:44Z

Looks cool. Thanks for tackling this. Look forward to trying it in a release. I'd like to consider swapping out our existing MFA caching solution to replace it with this. I tried to derive answers to some technical questions about this by reviewing the code changes but wasn't able to. If these get addressed in the release notes, please ignore and just ping me with a link to that when it ships.

some assumed questions / answers

How do you enable caching?

I assume aws-iam-authenticator --cached

Where does it store it's cached credentials on disk?

I assume somewhere in ~/.kube/config

What does it cache?

I assume something like this is cached on disk and that it lasts for 15 minutes before re-prompting for MFA. Yea?

{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1alpha1","spec":{},"status":{"token":"k8s-aws-v1.REDACTED...."}}`

llamahunter · 2019-05-02T18:52:32Z

See aws-iam-authenticator token --help. Note that this caching is for the SSO credentials used to retrieve the token, not the token itself. Someone else is working on token caching, I believe, but SSO credential fetching takes a long time (7-8 seconds), so caching the credentials is very useful.

Credentials are stored in ~/.kube/cache/aws-iam-authenticator/credentials.yaml unless you override the location with the AWS_IAM_AUTHENTICATOR_CACHE_FILE environment variable

It caches the access_key_id, secret_access_key, and session_token for as long as you have configured your credentials to be cacheable. I have mine set in IAM for 12 hours (the max).

truncs · 2019-05-02T21:25:05Z

@llamahunter Thanks for all the hard work on this! Much appreciated!
On the session duration, even after setting the max duration time in the IAM, the session durations that we get back are for 15 mins. Thoughts on why this might be the case? Happy to open a separate issue for this.

llamahunter · 2019-05-02T21:30:53Z

What is your SSO system? If using Okta, you must set the time in the ~/.okta/config.properties to set the OKTA_STS_DURATION. YMMV if using a different SSO system.

CharlieC3 · 2019-05-02T21:47:10Z

@truncs I don't believe the duration is configurable due to a lack of configuration options in the AWS API.
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html
#63
#75

llamahunter · 2019-05-02T22:25:19Z

Longer durations up to 12 hrs for federated access using roles for about a year now. https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/

CharlieC3 · 2019-05-02T22:33:48Z

@llamahunter According to the links in my previous comment, and comments from the other PRs, it seems like aws-iam-authenticator may be using the sig-v4-authenticating-requests method. And if it does, that would still be subject to the duration set by the sig-v4-authenticating-requests method, which is hard-set to 15 minutes.

So it's possible both duration timeouts are getting used here, but the shorter one is not configurable and is consequently the one we're being affected by.

Keep in mind I'm not familiar with this project's source code, but the description of the 15 minute timeout duration for the aws-iam-authenticator fits suspiciously well into the 15 minute auth timeout described in this page.
https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html

llamahunter · 2019-05-02T22:36:44Z

I think you're talking about token timeouts. This is about caching credentials, not tokens. I think there's another issue addressing the caching of tokens, for which there is a built in limit by AWS of 10-15 minutes.

llamahunter · 2019-05-02T22:38:30Z

See the list of 'possibly addresses in alternate way' issues listed at the start of this pull request.

CharlieC3 · 2019-05-02T22:43:51Z

@llamahunter You might be right, I was under the impression they were affected by the same issue described in this comment.
#63 (comment)

llamahunter · 2019-05-02T22:54:16Z

Yes, per their comment, they appear to be using Okta SSO 2FA stuff, like we do. If they set the IAM federated role connected to their Okta sign on to expire in 12 hours, and enable caching in aws-iam-authenticator, I suspect their lives will be a lot more pleasant. I believe aws-iam-authenticator still fetches the token every time, tho, which, in my experience, is a short time delay (< 1 second). It would be nice if the token were cached for the full 15 minutes to skip kubectl talking to anything other than the EKS k8s control plane endpoint on every invocation.

truncs · 2019-05-02T22:57:57Z

@llamahunter we use 2FA functionality provided by AWS.

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jan 11, 2019

k8s-ci-robot requested review from mattmoyer and nckturner January 11, 2019 01:28

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jan 11, 2019