AWS IAM role: Generate temporary security credentials from Role #350

SupriyaKasten · 2019-10-11T00:10:28Z

Change Overview

AWS IAM role: Generate temporary security credentials from Role
Earlier, we added the temp credentials to profile directly but since profiles are persistent we do not want them to get invalid after the session expires, hence the change.

Pull request type

Please check the type of change your PR introduces:

Issues

fix(tests): Fix flaky TestCloneGlobalLogger #2775

Test Plan

pkg/kando/location_test.go

 go test -v  -check.v -check.f TestLocationObjectStore
=== RUN   Test
PASS: location_test.go:35: LocationSuite.TestLocationObjectStore	3.188s
OK: 1 passed
--- PASS: Test (3.19s)
PASS
ok  	github.com/kanisterio/kanister/pkg/kando	3.227s

pkg/location/location_test.go

go test 
OK: 1 passed
PASS
ok  	github.com/kanisterio/kanister/pkg/location	2.242s

pkg/restic/restic_test.go

go test
OK: 10 passed
PASS
ok  	github.com/kanisterio/kanister/pkg/restic	0.715s

pkg/secrets/aws_test.go

go test -v  -check.v -check.f TestExtractAWSCredentialsWithSessionToken
=== RUN   Test
PASS: aws_test.go:84: AWSSecretSuite.TestExtractAWSCredentialsWithSessionToken	0.957s
OK: 1 passed
--- PASS: Test (0.96s)
PASS
ok  	github.com/kanisterio/kanister/pkg/secrets	0.980s

pkg/blockstorage/blockstorage_test.go Tested for auto refreshing temp credentials

 go test -timeout 30m
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1021] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1022] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1023] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1024] Update                                        State=creating Volume=vol-006b9ab4e0e4607bb
INFO[1027] Volume creation complete                      VolumeID=vol-006b9ab4e0e4607bb
INFO[1027] Snapshotting EBS volume                       volume_id=vol-006b9ab4e0e4607bb
INFO[1028] Snapshot completed                            SnapshotID=snap-0f3a5b6fb056d8e9d
INFO[1029] Ignoring duplicate tag                        RetainedValue= kanister.io/clustername=
INFO[1029] Ignoring duplicate tag                        RetainedValue= kanister.io/version=
INFO[1029] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1029] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1029] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1029] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1030] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1030] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1030] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1031] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1032] Update                                        State=creating Volume=vol-09b206f4faefb81ca
INFO[1035] Volume creation complete                      VolumeID=vol-09b206f4faefb81ca
INFO[1035] Deleting EBS Snapshot                         SnapshotID=snap-0f3a5b6fb056d8e9d
INFO[1035] Deleting EBS Snapshot                         SnapshotID=snap-0f3a5b6fb056d8e9d
INFO[1036] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1036] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1036] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1036] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1037] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1037] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1037] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1038] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1039] Update                                        State=creating Volume=vol-00ac807909b208cac
INFO[1042] Volume creation complete                      VolumeID=vol-00ac807909b208cac
INFO[1043] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1043] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1043] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1043] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1044] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1044] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1044] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1045] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1046] Update                                        State=creating Volume=vol-07bdf8d7c181f4413
INFO[1049] Volume creation complete                      VolumeID=vol-07bdf8d7c181f4413
INFO[1049] Snapshotting EBS volume                       volume_id=vol-07bdf8d7c181f4413
INFO[1051] Snapshot completed                            SnapshotID=snap-075a9388d739c2c40
INFO[1051] Deleting EBS Snapshot                         SnapshotID=snap-075a9388d739c2c40
INFO[1051] Deleting EBS Snapshot                         SnapshotID=snap-075a9388d739c2c40
OK: 3 passed, 1 skipped
PASS
ok  	github.com/kanisterio/kanister/pkg/blockstorage	1051.606s

Manual
Unit test
E2E

SupriyaKasten · 2019-10-11T00:10:55Z

pkg/restic/restic_test.go

-		{
-			profile: &param.Profile{
-				Location: v1alpha1.Location{
-					Type:     v1alpha1.LocationTypeS3Compliant,
-					Endpoint: "endpoint", // Also remove all of the trailing slashes
-				},
-				Credential: param.Credential{
-					Type: param.CredentialTypeSecret,
-					Secret: &v1.Secret{
-						Type: "secrets.kanister.io/aws",
-						Data: map[string][]byte{
-							"access_key_id":     []byte("id"),
-							"secret_access_key": []byte("secret"),
-							"session_token":     []byte("token"),
-						},
-					},
-				},
-			},
-			repo:     "repo",
-			password: "my-secret",
-			expected: []string{
-				"export AWS_ACCESS_KEY_ID=id\n",
-				"export AWS_SECRET_ACCESS_KEY=secret\n",
-				"export AWS_SESSION_TOKEN=token\n",
-				"export RESTIC_REPOSITORY=s3:endpoint/repo\n",
-				"export RESTIC_PASSWORD=my-secret\n",
-				"restic",
-			},
-		},


adding new test case in a followup commit!

SupriyaKasten · 2019-10-11T01:12:08Z

pkg/secrets/aws_test.go

-		{
-			secret: &v1.Secret{
-				Type: v1.SecretType(AWSSecretType),
-				Data: map[string][]byte{
-					AWSAccessKeyID:     []byte("key_id"),
-					AWSSecretAccessKey: []byte("secret_key"),
-					AWSSessionToken:    []byte("session_token"),
-				},
-			},
-			expected: &credentials.Value{
-				AccessKeyID:     "key_id",
-				SecretAccessKey: "secret_key",
-				SessionToken:    "session_token",
-			},
-			errChecker: IsNil,
-		},


TODO: add unit test

Added unit test below.

hakanmemisoglu · 2019-10-11T18:18:08Z

pkg/config/aws/aws.go

+		if err != nil {
+			return nil, "", "", errors.Wrap(err, "Failed to get temporary security credentials")
+		}
+		return config, region, role, nil


Do we need to return role here even after assuming role?

SupriyaKasten · 2019-10-21T22:28:51Z

pkg/config/aws/aws.go

@@ -33,23 +36,39 @@ const (
 	SecretAccessKey = "AWS_SECRET_ACCESS_KEY"
 	// SessionToken represents AWS Session Key
 	SessionToken = "AWS_SESSION_TOKEN"
+
+	assumeRoleDuration = 25 * time.Minute


For Blockstorage 👆

The assumption being that we'll automatically renew the blockstorage creds, right?

SupriyaKasten · 2019-10-21T22:29:59Z

pkg/secrets/aws.go

+	// ConfigRole represents the key for the ARN of the role which can be assumed.
+	// It is optional.
+	ConfigRole         = "role"
+	assumeRoleDuration = 90 * time.Minute


For Restic and Kando 👆

tdmanv

Please include examples of running the tests you changes, especailly the ones that won't get run in CI due to the AWS key requirement.

pkg/config/aws/aws.go

tdmanv · 2019-10-21T23:12:05Z

pkg/config/aws/aws.go

 	region, ok := config[ConfigRegion]
 	if !ok {
-		return nil, "", "", errors.New("region required for storage type EBS")
+		return nil, "", errors.New("region required for storage type EBS")


Is this EBS specific?

tdmanv · 2019-10-21T23:12:57Z

pkg/config/aws/role.go

+
+// SwitchRole assumes role with given credentials.
+// Temporary credentials will be valid for 'duration'.
+func SwitchRole(ctx context.Context, accessKeyID string, secretAccessKey string, role string, duration time.Duration) (*credentials.Credentials, error) {


This doesn't really describe the behavior. creds Is a creds provider that will automatically refresh credentials.

pkg/config/aws/role.go

Co-Authored-By: Thomas Manville <tom@kasten.io>

SupriyaKasten · 2019-10-22T00:29:13Z

Updated the commit description with test results.

tdmanv · 2019-10-22T00:51:29Z

pkg/config/helpers.go

+	"gopkg.in/check.v1"
+)
+
+func GetEnvOrSkip(c *check.C, varName string) string {


Is this method already defined in testutil?

https://github.com/kanisterio/kanister/blob/master/pkg/testutil/fixture.go#L70

@tdmanv Yes, It creates a cycle if I import pkg/testutil, hence I added the function to helpers.go
Also, this func is defined in 4 different places in kanister 😄 . Once this PR is merged, I can push another followup PR to remove the duplicate funcs and import only from helpers.go.

that works for now

hakanmemisoglu · 2019-10-22T16:37:52Z

pkg/config/aws/aws.go

 )

 // GetConfig returns a configuration to establish AWS connection, connected region name and the role to assume if it exists.
-func GetConfig(config map[string]string) (awsConfig *aws.Config, region string, role string, err error) {
+func GetConfig(ctx context.Context, config map[string]string) (awsConfig *aws.Config, region string, err error) {


This doc might also need update. It doesn't return the role but assumes it if it exists in config.

Updated the comment!

tdmanv · 2019-10-22T21:42:45Z

pkg/config/aws/aws.go

@@ -33,23 +36,39 @@ const (
 	SecretAccessKey = "AWS_SECRET_ACCESS_KEY"
 	// SessionToken represents AWS Session Key
 	SessionToken = "AWS_SESSION_TOKEN"
+
+	assumeRoleDuration = 25 * time.Minute


The assumption being that we'll automatically renew the blockstorage creds, right?

SupriyaKasten · 2019-10-23T18:17:40Z

pkg/blockstorage/blockstorage_test.go

@@ -272,6 +272,7 @@ func (s *BlockStorageProviderSuite) getConfig(c *C, region string) map[string]st
 		}
 		config[awsconfig.AccessKeyID] = accessKey
 		config[awsconfig.SecretAccessKey] = secretAccessKey
+		config[awsconfig.ConfigRole] = os.Getenv(awsconfig.ConfigRole)


@tdmanv this enables the tests with AWS assume role BUT it fails for me locally with the same error as we saw with before for storage_tests INFO[0000] Failed to create volume error="UnauthorizedOperation: You are not authorized to perform this operation. Encoded authorization failure message

Tested successfully with different role.

SupriyaKasten · 2019-10-23T18:22:26Z

pkg/objectstore/objectstore_test.go

+		awsAccessKeyID := os.Getenv("AWS_ACCESS_KEY_ID")
+		awsSecretAccessKey := os.Getenv("AWS_SECRET_ACCESS_KEY")
+		var awsSessionToken string
+		if role, ok := os.LookupEnv(aws.ConfigRole); ok {
+			creds, err := aws.SwitchRole(ctx, awsAccessKeyID, awsSecretAccessKey, role, assumeRoleDuration)
+			c.Check(err, IsNil)
+			val, err := creds.Get()
+			c.Check(err, IsNil)
+			awsAccessKeyID = val.AccessKeyID
+			awsSecretAccessKey = val.SecretAccessKey
+			awsSessionToken = val.SessionToken
+		}


@tdmanv With these changes, We run objectstore tests with AWS IAM role, one of the 8 tests fails with my credentials:

go test ---------------------------------------------------------------------- FAIL: objectstore_test.go:423: ObjectStoreProviderSuite.TestBucketGetRegions objectstore_test.go:429: c.Check(err, IsNil) ... value *errors.withStack = Containers, listing the buckets: AccessDenied: Access Denied status code: 403, request id: FC39088EE1A52416, host id: g9XaZ2ITdRbK4rDjjR+MCkv8sGlS8lLvubXbq4jnOZRyB/CQU7e0Gk/hyKW86wvffTo4EcQjO4M= ("Containers, listing the buckets: AccessDenied: Access Denied\n\tstatus code: 403, request id: FC39088EE1A52416, host id: g9XaZ2ITdRbK4rDjjR+MCkv8sGlS8lLvubXbq4jnOZRyB/CQU7e0Gk/hyKW86wvffTo4EcQjO4M=") objectstore_test.go:439: c.Check(err, IsNil) ... value *errors.withStack = Containers, listing the buckets: AccessDenied: Access Denied status code: 403, request id: B02BD08131EADD44, host id: UTVEUGHfLkhv6ZUs07OHvc8D74LSMMT4sw5ZBQecbjkqVtf55jteT3+TSytkhxgQ1yHGBe6LNK4= ("Containers, listing the buckets: AccessDenied: Access Denied\n\tstatus code: 403, request id: B02BD08131EADD44, host id: UTVEUGHfLkhv6ZUs07OHvc8D74LSMMT4sw5ZBQecbjkqVtf55jteT3+TSytkhxgQ1yHGBe6LNK4=") objectstore_test.go:439: c.Check(err, IsNil) ... value *errors.withStack = Containers, listing the buckets: AccessDenied: Access Denied status code: 403, request id: A34CE00D4BC213E7, host id: BtgIfwhLgvf+WCZDOvi6jG8zg63XVaHp1f4jUn8UL0qUMHyuFDTJhqqB4wXoiwnooF0NSy90zfE= ("Containers, listing the buckets: AccessDenied: Access Denied\n\tstatus code: 403, request id: A34CE00D4BC213E7, host id: BtgIfwhLgvf+WCZDOvi6jG8zg63XVaHp1f4jUn8UL0qUMHyuFDTJhqqB4wXoiwnooF0NSy90zfE=") objectstore_test.go:439: c.Check(err, IsNil) ... value *errors.withStack = Containers, listing the buckets: AccessDenied: Access Denied status code: 403, request id: 9017B3A3E89EA391, host id: 6+JXjooSmq1FMlitzW787EsZSz8ktC3DWfvC64wRvsvBhrgvkKi9ikCgi2PpqR4KbeblH6KAq4w= ("Containers, listing the buckets: AccessDenied: Access Denied\n\tstatus code: 403, request id: 9017B3A3E89EA391, host id: 6+JXjooSmq1FMlitzW787EsZSz8ktC3DWfvC64wRvsvBhrgvkKi9ikCgi2PpqR4KbeblH6KAq4w=") objectstore_test.go:439: c.Check(err, IsNil) ... value *errors.withStack = Containers, listing the buckets: AccessDenied: Access Denied status code: 403, request id: 9B88820B15FB881A, host id: 8oyRbM9NywaXjgQYISX3j1n+cTS3okFPb32nMt3fpjKXNH8TzwzcjfbQVYEj9DuNHQLwIWmZPjQ= ("Containers, listing the buckets: AccessDenied: Access Denied\n\tstatus code: 403, request id: 9B88820B15FB881A, host id: 8oyRbM9NywaXjgQYISX3j1n+cTS3okFPb32nMt3fpjKXNH8TzwzcjfbQVYEj9DuNHQLwIWmZPjQ=") OOPS: 7 passed, 1 skipped, 1 FAILED --- FAIL: Test (8.70s) FAIL exit status 1 FAIL github.com/kanisterio/kanister/pkg/objectstore 8.729s

@SupriyaKasten is this failing before or after we get temporary credentials?

These fail after we get the temp creds, although 7 of the tests pass with the temp credentials. The call to s.provider.ListBuckets(ctx) fails with these temp credentials.

If so, then this is an issue with the Role we're changing to in these tests, so won't this fail if we run this test with anyone's creds?

Discussed offline with @tdmanv and @vkamra, skipping the failing test since we want to run the tests with available role permissions(our s3 role does not have list bucket permission)

tdmanv · 2019-10-23T22:30:36Z

pkg/blockstorage/awsebs/awsebs.go

@@ -406,6 +441,10 @@ func setResourceTags(ctx context.Context, ec2Cli *EC2, resourceID string, tags m
 }

 func (s *ebsStorage) VolumeCreateFromSnapshot(ctx context.Context, snapshot blockstorage.Snapshot, tags map[string]string) (*blockstorage.Volume, error) {
+	err := s.refreshCreds()


The credentials should be automatically refreshed by the EC2 client

vkamra · 2019-10-24T05:38:46Z

pkg/objectstore/objectstore_test.go

-var _ = Suite(&ObjectStoreProviderSuite{osType: ProviderTypeGCS, region: ""})
-var _ = Suite(&ObjectStoreProviderSuite{osType: ProviderTypeAzure, region: ""})
+
+// var _ = Suite(&ObjectStoreProviderSuite{osType: ProviderTypeGCS, region: ""})


Uncomment prior to merge

AWS IAM: Generate Temp creds from Role

34d1960

SupriyaKasten commented Oct 11, 2019

View reviewed changes

SupriyaKasten requested review from hakanmemisoglu and tdmanv October 11, 2019 00:11

SupriyaKasten changed the title ~~AWS IAM role: Generate temporary security credentials from Role~~ (WIP)AWS IAM role: Generate temporary security credentials from Role Oct 11, 2019

SupriyaKasten added 2 commits October 10, 2019 17:47

fix error

96c0731

remove unit test for role since it fails

a577fe7

SupriyaKasten commented Oct 11, 2019

View reviewed changes

hakanmemisoglu reviewed Oct 11, 2019

View reviewed changes

SupriyaKasten added 5 commits October 14, 2019 17:42

Add unit test for restic

c0b4e5e

more tests and refactoring

cc75178

Merge branch 'master' into aws-iam-profile

09215b2

fix errors

fa01826

Adding the missing file & fix error

a26dbd9

SupriyaKasten force-pushed the aws-iam-profile branch from b96630d to a26dbd9 Compare October 17, 2019 00:07

IAM role support for blockstorage + refactoring

238d64f

SupriyaKasten added the hold off merging label Oct 21, 2019

SupriyaKasten changed the title ~~(WIP)AWS IAM role: Generate temporary security credentials from Role~~ AWS IAM role: Generate temporary security credentials from Role Oct 21, 2019

SupriyaKasten added 2 commits October 21, 2019 15:10

remove static credential creation

4378f8e

create test profile with role

24fab41

SupriyaKasten commented Oct 21, 2019

View reviewed changes

tdmanv reviewed Oct 21, 2019

View reviewed changes

SupriyaKasten and others added 3 commits October 21, 2019 16:32

Update pkg/config/aws/role.go

2c3724b

Co-Authored-By: Thomas Manville <tom@kasten.io>

Update pkg/config/aws/aws.go

a3ba79e

Co-Authored-By: Thomas Manville <tom@kasten.io>

Address review suggestions

8578737

tdmanv reviewed Oct 22, 2019

View reviewed changes

hakanmemisoglu reviewed Oct 22, 2019

View reviewed changes

Trivial:Update comment

c110f29

Switch role in objectstore if env variable role set

1ed5162

tdmanv approved these changes Oct 22, 2019

View reviewed changes

SupriyaKasten added 4 commits October 22, 2019 16:50

refresh aws blockstorage creds + avoid objectsore errors

ea2434f

Merge branch 'master' into aws-iam-profile

c2575ba

nit: import time

0a40baa

Some more unit test

ded9a6e

SupriyaKasten commented Oct 23, 2019

View reviewed changes

fix ci errors

161b297

SupriyaKasten force-pushed the aws-iam-profile branch from 00b1445 to 161b297 Compare October 23, 2019 18:44

tdmanv reviewed Oct 23, 2019

View reviewed changes

SupriyaKasten added 2 commits October 23, 2019 18:16

Skip objectsotre_test for listing bucket

49849ab

remove refresh creds func

99283d9

vkamra reviewed Oct 24, 2019

View reviewed changes

SupriyaKasten added 2 commits October 24, 2019 10:01

uncomment tests

f112f08

nit

bc2a7ff

tdmanv approved these changes Oct 24, 2019

View reviewed changes

SupriyaKasten removed the hold off merging label Oct 24, 2019

Merge branch 'master' into aws-iam-profile

2f38287

mergify bot merged commit 2733489 into master Oct 24, 2019

mergify bot deleted the aws-iam-profile branch October 24, 2019 20:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

AWS IAM role: Generate temporary security credentials from Role #350

AWS IAM role: Generate temporary security credentials from Role #350

SupriyaKasten commented Oct 11, 2019 •

edited

Loading

SupriyaKasten Oct 11, 2019

SupriyaKasten Oct 11, 2019

SupriyaKasten Oct 21, 2019

hakanmemisoglu Oct 11, 2019

SupriyaKasten Oct 21, 2019

tdmanv Oct 22, 2019

SupriyaKasten Oct 21, 2019

tdmanv left a comment

tdmanv Oct 21, 2019

tdmanv Oct 21, 2019

SupriyaKasten commented Oct 22, 2019

tdmanv Oct 22, 2019

tdmanv Oct 22, 2019

SupriyaKasten Oct 22, 2019

tdmanv Oct 22, 2019

hakanmemisoglu Oct 22, 2019

SupriyaKasten Oct 22, 2019

tdmanv Oct 22, 2019

SupriyaKasten Oct 23, 2019

SupriyaKasten Oct 24, 2019

SupriyaKasten Oct 23, 2019

tdmanv Oct 23, 2019

SupriyaKasten Oct 23, 2019 •

edited

Loading

tdmanv Oct 23, 2019

SupriyaKasten Oct 24, 2019

tdmanv Oct 23, 2019

vkamra Oct 24, 2019

AWS IAM role: Generate temporary security credentials from Role #350

AWS IAM role: Generate temporary security credentials from Role #350

Conversation

SupriyaKasten commented Oct 11, 2019 • edited Loading

Change Overview

Pull request type

Issues

Test Plan

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tdmanv left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SupriyaKasten commented Oct 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SupriyaKasten Oct 23, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SupriyaKasten commented Oct 11, 2019 •

edited

Loading

SupriyaKasten Oct 23, 2019 •

edited

Loading