Kops doesn't respect AWS_PROFILE when using ROLE_ARN

[jmcclell]

1. What kops version are you running? The command kops version, will display
this information.

Version 1.10.0 (git-8b52ea6d1)

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

Irrelevant.

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

Given an ec2 instance in account-A with an instance profile role with permissions to assume an admin role in account-B, we setup ~/.aws/credentials such that

[account-b-admin]
role_arn=$CROSS_ACCOUNT_ADMIN_ROLE_IN_ACCOUNT_B_ARN
credential_source=Ec2InstanceMetadata
region=us-east-1

We then create an S3 bucket in account-b called my-kops-state, henceforth referenced as $KOPS_STATE_STORE

Then, set the appropriate AWS ENV variables:

$ export AWS_PROFILE=account-b-admin
$ export AWS_DEFAULT_PROFILE=account-b-admin # covering all bases
$ export AWS_SDK_LOAD_CONFIG=1 # because go can be weird

At this point, using the AWS CLI, we can verify S3 bucket connectivity.

$aws s3 ls $KOPS_STATE_STORE
# no errors. the bucket is empty and listable.

$ aws s3api get-bucket-location --bucket $KOPS_STATE_STORE
{
    "LocationConstraint": null
}

However, running even the simplest kops command will fail due to s3 access issues.

$ kops create cluster --name=my-cluster --state=$KOPS_STATE_STORE

error reading cluster configuration "my-cluster": error reading s3://my-kops-state/my-cluster/config: Could not retrieve location for AWS bucket my-kops-state

5. What happened after the commands executed?

Kops fails to access the s3 bucket, meaning it's clearly using the wrong credentials.

6. What did you expect to happen?

Kops should use the same credentials as the AWS CLI, set via the AWS_PROFILE environment variable, and thus be able to access the state bucket.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

N/A

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

kops create cluster --name=$NAME --state=$KOPS_STATE_STORE -v10
I1006 12:52:30.109512   18492 factory.go:68] state store s3://$KOPS_STATE_STORE
I1006 12:52:30.413189   18492 s3context.go:162] unable to get bucket location from region "us-east-1"; scanning all regions: AccessDenied: Access Denied
        status code: 403, request id: 5A8270D729D08557, host id: S5o9WrcXYdpvElXCGh/TUOAWx4Pu4eE1vekRUb1EMmXdSStdn1QBKNoQC1pYwcw7gt2FMGJAmqw=
I1006 12:52:30.463520   18492 s3context.go:250] Querying S3 for bucket location for $KOPS_STATE_STORE
I1006 12:52:30.463562   18492 s3context.go:255] Doing GetBucketLocation in "us-west-2"
I1006 12:52:30.463569   18492 s3context.go:255] Doing GetBucketLocation in "sa-east-1"
I1006 12:52:30.463581   18492 s3context.go:255] Doing GetBucketLocation in "eu-central-1"
I1006 12:52:30.463601   18492 s3context.go:255] Doing GetBucketLocation in "us-east-2"
I1006 12:52:30.463607   18492 s3context.go:255] Doing GetBucketLocation in "us-west-1"
I1006 12:52:30.463627   18492 s3context.go:255] Doing GetBucketLocation in "ap-southeast-1"
I1006 12:52:30.463636   18492 s3context.go:255] Doing GetBucketLocation in "ca-central-1"
I1006 12:52:30.463647   18492 s3context.go:255] Doing GetBucketLocation in "eu-west-2"
I1006 12:52:30.463662   18492 s3context.go:255] Doing GetBucketLocation in "ap-northeast-2"
I1006 12:52:30.463680   18492 s3context.go:255] Doing GetBucketLocation in "us-east-1"
I1006 12:52:30.463692   18492 s3context.go:255] Doing GetBucketLocation in "eu-west-1"
I1006 12:52:30.463773   18492 s3context.go:255] Doing GetBucketLocation in "eu-west-3"
I1006 12:52:30.463713   18492 s3context.go:255] Doing GetBucketLocation in "ap-southeast-2"
I1006 12:52:30.463741   18492 s3context.go:255] Doing GetBucketLocation in "eu-north-1"
I1006 12:52:30.463853   18492 s3context.go:255] Doing GetBucketLocation in "ap-northeast-1"
I1006 12:52:30.463734   18492 s3context.go:255] Doing GetBucketLocation in "ap-south-1"

error reading cluster configuration "$NAME": error reading s3://$KOPS_STATE_STORE/$NAME/config: Could not retrieve location for AWS bucket $KOPS_STATE_STORE

9. Anything else do we need to know?

All other tools using the AWS SDK (including Go) are working just fine with this setup. The role we are assuming has full admin privileges to the account owning the bucket.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[jmcclell]

/remove-lifecycle stale

Still an issue.

[mattclegg]

The problem here is that support for credential_source is only added in aws-sdk in a later release than the version of KOPS your using; aws/aws-sdk-go#2201

I needed this feature in 1.11 and have backported a patch (https://gist.github.com/mattclegg/af6e71c4934d91772ab8b6a72ac225a1). It should be straight forward to also do for 1.10

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gauravpatel2016]

Same issue here

[alebsack]

I also see this using KOPS 1.18.

/reopen

[Sami-svmx]

same issue with Kops 1.18

[olegsa]

try to setup environment varieble AWS_SDK_LOAD_CONFIG
export AWS_SDK_LOAD_CONFIG=1

[david-seo-dubber]

try to setup environment varieble AWS_SDK_LOAD_CONFIG
export AWS_SDK_LOAD_CONFIG=1

This worked for me and I am on Kops 1.14. If anyone is using arn roles to assume into the profile please use this and give it a go.