AWS credentials not handled correctly

[jbergknoff-rival]

There are various issues with the AWS credential handling when downloading from S3. #185 #157 #152 #22 gruntwork-io/terragrunt#775

I'm less familiar with the AWS SDK for Go, but in my experience with those for JavaScript and Python, the credential handling here [1] looks like it can be made simpler and more robust by using the SDK as documented [2].

The particulars about communicating with the EC2 (or ECS) metadata API, reading the shared config or credentials file, choosing a region, etc., are all handled by the SDK's default credential provider chain. The explicit provider chain constructed in getAWSConfig lacks support for various edge cases.

Would you be open to a PR changing the way AWS sessions are created?

[1]

go-getter/get_s3.go

Lines 182 to 217 in da0323b

	func (g *S3Getter) getAWSConfig(region string, url *url.URL, creds *credentials.Credentials) *aws.Config {
	conf := &aws.Config{}
	if creds == nil {
	// Grab the metadata URL
	metadataURL := os.Getenv("AWS_METADATA_URL")
	if metadataURL == "" {
	metadataURL = "http://169.254.169.254:80/latest"
	}
	creds = credentials.NewChainCredentials(
	[]credentials.Provider{
	&credentials.EnvProvider{},
	&credentials.SharedCredentialsProvider{Filename: "", Profile: ""},
	&ec2rolecreds.EC2RoleProvider{
	Client: ec2metadata.New(session.New(&aws.Config{
	Endpoint: aws.String(metadataURL),
	})),
	},
	})
	}
	if creds != nil {
	conf.Endpoint = &url.Host
	conf.S3ForcePathStyle = aws.Bool(true)
	if url.Scheme == "http" {
	conf.DisableSSL = aws.Bool(true)
	}
	}
	conf.Credentials = creds
	if region != "" {
	conf.Region = aws.String(region)
	}
	return conf
	}

[2] "Configuring Credentials" section on https://docs.aws.amazon.com/sdk-for-go/api/

[blockmar]

+1

I can verify that there still are issues with how Role based authentications are handled in go-getter:

Fails:

$ ./go-getter mybucket.s3-eu-north-1.amazonaws.com/path/my-file.jar my-file.jar 2019/07/05 12:57:01 Error downloading: AccessDenied: Access Denied
	status code: 403, request id: 18712XXXXCBE2CED, host id: iSKc/1xteeXXXXXXXXXXXXXXXs9ieOT2WjY2Q41YJYZFCpwG/E2P82bgmPANA=

Works:

$ aws s3 cp s3://mybucket/path/my-file.jar my-file.jar
download: s3://mybucket/path/myfile.jar to ./myfile.jar

(No AWS credentials set in environment nor credentials file)

Tested using master of go-getter from a server in us-east-1 and a bucket in eu-north-1.

[bashims]

I am working on a PR for this as we speak. The lack of support AWS profiles and STS (AssumeRole) is showstopper for us. Using the default credential providers should work just fine here.

[solarmosaic-kflorence]

Please fix this.