Issue upgrading from v0.4.0 to v0.5.0

[chinthakagodawita]

Hi,

We're running a kops cluster with a manually installed version of v0.4.0 which has been working with no issues.

We're attempting an upgrade on to v0.5.0 (specifically v0.5.0-alpine-3.7) with the following server arguments:

      - name: aws-iam-authenticator
        image: [...]:v0.5.0-alpine-3.7
        args:
        - server
        - --config=/etc/aws-iam-authenticator/config.yaml
        - --state-dir=/var/aws-iam-authenticator
        - --generate-kubeconfig=/etc/kubernetes/aws-iam-authenticator/kubeconfig.yaml
        # only for kops (https://github.com/goodfoo/aws-iam-authenticator#kops-usage)
        - --kubeconfig-pregenerated=true
        - --backend-mode=ConfigMap,CRD,File

I added backend-mode flag so that it uses our existing configmap, prior to adding this the daemonset did not log anything past waiting for informer caches to sync (logs below).

Post-upgrade, all kubectl commands fail with: error: You must be logged in to the server (Unauthorized).

Token verification appears to work, i.e. running:

• aws-iam-authenticator token -i CLUSTER
• aws-iam-authenticator verifiy -i CLUSTER -t TOKEN

The daemonset's logs have the below, with the final log line repeating:

time="2020-02-04T12:24:06Z" level=info msg="creating event broadcaster"
time="2020-02-04T12:24:06Z" level=info msg="setting up event handlers"
time="2020-02-04T12:24:06Z" level=info msg="starting mapper \"ConfigMap\""
time="2020-02-04T12:24:06Z" level=info msg="starting mapper \"CRD\""
time="2020-02-04T12:24:06Z" level=info msg="starting aws iam authenticator controller"
time="2020-02-04T12:24:06Z" level=info msg="waiting for informer caches to sync"
time="2020-02-04T12:24:06Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:11Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:16Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:21Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:26Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:31Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:36Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds"
time="2020-02-04T12:24:41Z" level=warning msg="Unable to re-establish watch.  Sleeping for 5 seconds" 

Are we missing an upgrade step? I've tried regenerating the certs that kops uses and rolling the masters but nothing seems to help.

[nckturner]

/assign @wongma7

[wongma7]

@chinthakagodawita I think what you want is File,CRD. When you say "existing configmap", do you mean the configmap that's mounted into the pod? Because mode ConfigMap refers to the EKS method of watching a ConfigMap which has only ever existed in EKS whereas File refers to the statically mounted file (which is also a configmap...not the best naming.) The documentation isn't very explicit about what the values for this flag mean exactly, let me fix it https://github.com/kubernetes-sigs/aws-iam-authenticator#configure-iamidentitymapping-custom-resource-definitions

However, either way you should not be getting Unauthorized errors because it's supposed to fall back in the order you specified. Do you have more logs from the authenticator showing 'access denied' errors

[chinthakagodawita]

@wongma7 That might be it. I should be able to give it another test in the next week or two, will update you once I know more.

[rifelpet]

I'm working on upgrading Kops' integration with aws-iam-authenticator and am running into a similar issue. the new --backend-mode flag description states:

The first one that returns a matching mapping wins

My cluster does not have an EKS ConfigMap or any IAMIdentityMapping resources, only the ConfigMap volume mount like what was required before 0.5.0. Without specifying --backend-mode, aws-iam-authenticator chooses to use the CRD mapping and subsequently doesn't detect any mappings, so all authn fails. I have to explicitly set --backend-mode=File in order to preserve the pre-0.5.0 behavior. This makes me think the flag's description is not accurate, or we could improve the logic of how each backend mode is initialized and used.

Logs without --backend-mode=File:

time="2020-02-28T13:22:29Z" level=info msg="creating event broadcaster"
time="2020-02-28T13:22:29Z" level=info msg="setting up event handlers"
time="2020-02-28T13:22:29Z" level=info msg="starting mapper \"CRD\""
time="2020-02-28T13:22:29Z" level=info msg="starting aws iam authenticator controller"
time="2020-02-28T13:22:29Z" level=info msg="waiting for informer caches to sync"

and then no further logs from any of the failed authn requests.

Logs with --backend-mode=File:

time="2020-02-28T13:24:29Z" level=info msg="mapping IAM role" groups="[foo]" role="arn:aws:iam::000000000000:role/foo" username="foo:{{SessionName}}"
time="2020-02-28T13:24:29Z" level=info msg="loaded existing keypair" certPath=/var/aws-iam-authenticator/cert.pem keyPath=/var/aws-iam-authenticator/key.pem
time="2020-02-28T13:24:29Z" level=info msg="listening on https://127.0.0.1:21362/authenticate"
time="2020-02-28T13:24:29Z" level=info msg="reconfigure your apiserver with `--authentication-token-webhook-config-file=/etc/kubernetes/aws-iam-authenticator/kubeconfig.yaml` to enable (assuming default hostPath mounts)"
time="2020-02-28T13:25:04Z" level=info msg="STS response" accesskeyid=xxx accountid=000000000000 arn="arn:aws:sts::000000000000:assumed-role/foo" client="127.0.0.1:60230" method=POST path=/authenticate session=foo userid=xxx
time="2020-02-28T13:25:04Z" level=info msg="access granted" arn="arn:aws:iam::000000000000:role/foo" client="127.0.0.1:60230" groups="[foo]" method=POST path=/authenticate uid="aws-iam-authenticator:000000000000:xxx" username="foo"

Is this expected? I know this project is pre 1.0 release so breaking changes can be expected, but

1. the flag's description seems to be inaccurate
2. it'd be great if a required change like this was prominently mentioned in the release notes

I'm happy to open a PR if we can agree on the expected behavior and flag description.

[chinthakagodawita]

I can confirm @rifelpet's findings. Looks like that backend-mode flag was it.

When I run v0.5.0 with --backend-mode=ConfigMap,CRD,File, I get the same logs as I posted above with the same authentication errors. I still get authentication errors if I run it with --backend-mode=File,CRD,ConfigMap.

If however, I run with just --backend-mode=File, I get the below logs and kubectl is able to authenticate successfully:

time="2020-03-04T10:21:15Z" level=info msg="starting mapper \"File\""
time="2020-03-04T10:21:15Z" level=info msg="mapping IAM role" groups="[system:masters]" role="arn:aws:iam::XXX:role/ROLE_NAME" username=kubernetes-admin
time="2020-03-04T10:21:15Z" level=info msg="mapping IAM role" groups="[system:bootstrappers aws:instances]" role="arn:aws:iam::XXX:role/KubernetesNode" username="aws:{{AccountID}}:instance:{{SessionName}}"
time="2020-03-04T10:21:15Z" level=info msg="mapping IAM role" groups="[system:masters]" role="arn:aws:iam::XXX:role/ROLE_NAME" username="admin:{{SessionName}}"
time="2020-03-04T10:21:15Z" level=info msg="loaded existing keypair" certPath=/var/aws-iam-authenticator/cert.pem keyPath=/var/aws-iam-authenticator/key.pem
time="2020-03-04T10:21:15Z" level=info msg="listening on https://127.0.0.1:21362/authenticate"
time="2020-03-04T10:21:15Z" level=info msg="reconfigure your apiserver with `--authentication-token-webhook-config-file=/etc/kubernetes/aws-iam-authenticator/kubeconfig.yaml` to enable (assuming default hostPath mounts)"
time="2020-03-04T10:22:11Z" level=info msg="STS response" accesskeyid=ACCESS_KEY_ID accountid=XXX arn="arn:aws:sts::XXX:assumed-role/ROLE_NAME/TS" client="127.0.0.1:60564" method=POST path=/authenticate session=TS userid=USER_ID
time="2020-03-04T10:22:11Z" level=info msg="access granted" arn="arn:aws:iam::XXX:role/ROLE_NAME" client="127.0.0.1:60564" groups="[system:masters]" method=POST path=/authenticate uid="aws-iam-authenticator:XXX:USER_ID" username="admin:TS"

[wongma7]

I've opened a PR to fix this, please take a look at https://github.com/kubernetes-sigs/aws-iam-authenticator/pull/303/files#diff-04c6e90faac2675aa89e2176d2eec7d8R85

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[chinthakagodawita]

Looks like #303 is merged and is part of v0.5.1-alpha.1, thanks @wongma7!