Can't use `optFn` to set region for S3 Express operations (credentials issue) #2777

stefansundin · 2024-09-06T18:07:43Z

Acknowledgements

I have searched (https://github.com/aws/aws-sdk/issues?q=is%3Aissue) for past instances of this issue
I have verified all of my SDK modules are up-to-date (you can perform a bulk update with go get -u github.com/aws/aws-sdk-go-v2/...)

Describe the bug

S3 Express buckets require credentials that are implicitly retrieved with the CreateSession API operation. However, if the region is set using an optFn, that region is not propagated to the CreateSession call, and results in this error:

operation error S3: GetObject, get identity: get credentials: operation error S3: CreateSession, https response error StatusCode: 0, RequestID: , HostID: , request send failed, Get "https://my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com/?session=": dial tcp: lookup my-test-bucket--usw2-az1--x-s3.s3express-usw2-az1.us-west-1.amazonaws.com: no such host

In this case my S3 client was initialized with the us-west-1 region, but I'm attempting to set the region to us-west-2 in an optFn, but this is not respected in the CreateSession call.

Expected Behavior

I expect optFn functions to be forwarded to the CreateSession operation.

Current Behavior

optFn functions do not affect S3 Express credentials fetching.

Reproduction Steps

package main

import (
	"context"
	"fmt"
	"os"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/s3"
)

func main() {
	bucket := "my-test-bucket--usw2-az1--x-s3" // this S3 Express bucket is in us-west-2
	key := "test"
	initialRegion := "us-west-1" // incorrect region for my S3 Express bucket
	debug := true

	// I want to set the correct region using an optFn when making the GetObject call:
	setRegionFn := func(o *s3.Options) {
		o.Region = "us-west-2"
	}

	// Initialize the AWS SDK
	cfg, err := config.LoadDefaultConfig(
		context.TODO(),
		config.WithRegion(initialRegion),
		func(o *config.LoadOptions) error {
			if debug {
				var lm aws.ClientLogMode = aws.LogRequest | aws.LogResponse
				o.ClientLogMode = &lm
			}
			return nil
		},
	)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	client := s3.NewFromConfig(cfg)

	obj, err := client.GetObject(context.TODO(), &s3.GetObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}, setRegionFn) // <-- optFn used here

	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(1)
	}

	fmt.Fprintf(os.Stderr, "GetObject ContentLength: %d\n", aws.ToInt64(obj.ContentLength))
}

Possible Solution

Apply optFn functions first, and then clone the resulting client options before calling CreateSession?

Additional Information/Context

No response

AWS Go SDK V2 Module Versions Used

example github.com/aws/aws-sdk-go-v2@v1.30.5
example github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4
example github.com/aws/aws-sdk-go-v2/config@v1.27.33
example github.com/aws/aws-sdk-go-v2/credentials@v1.17.32
example github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
example github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
example github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
example github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1
example github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17
example github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
example github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19
example github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
example github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17
example github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2
example github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
example github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
example github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
example github.com/aws/smithy-go@v1.20.4
example go@1.23.1
github.com/aws/aws-sdk-go-v2@v1.30.5 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2@v1.30.5 github.com/jmespath/go-jmespath@v0.4.0
github.com/aws/aws-sdk-go-v2@v1.30.5 go@1.21
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4 go@1.21
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/credentials@v1.17.32
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/config@v1.27.33 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/config@v1.27.33 go@1.21
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/credentials@v1.17.32 go@1.21
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/feature/ec2/imds@v1.16.13 go@1.21
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17 go@1.21
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17 go@1.21
github.com/aws/aws-sdk-go-v2/internal/ini@v1.8.1 go@1.21
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19 go@1.21
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17 go@1.21
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream@v1.6.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/internal/v4a@v1.3.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/checksum@v1.3.19
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/aws-sdk-go-v2/service/internal/s3shared@v1.17.17
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/s3@v1.61.2 go@1.21
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/sso@v1.22.7 go@1.21
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/ssooidc@v1.26.7 go@1.21
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2@v1.30.5
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/internal/configsources@v1.3.17
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/internal/endpoints/v2@v2.6.17
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding@v1.11.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/aws-sdk-go-v2/service/internal/presigned-url@v1.11.19
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 github.com/aws/smithy-go@v1.20.4
github.com/aws/aws-sdk-go-v2/service/sts@v1.30.7 go@1.21
github.com/aws/smithy-go@v1.20.4 go@1.21
go@1.23.1 toolchain@go1.23.1

Compiler and Version used

go version go1.23.1 darwin/arm64

Operating System and version

macOS 14.6.1

The text was updated successfully, but these errors were encountered:

lucix-aws · 2024-09-06T19:10:15Z

Agreed. Workaround for the time being is to have multiple clients per region / config.

stefansundin added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024

lucix-aws added p2 This is a standard priority issue queued This issues is on the AWS team's backlog and removed needs-triage This issue or PR still needs to be triaged. labels Sep 6, 2024

lucix-aws added the workaround-available label Sep 6, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't use `optFn` to set region for S3 Express operations (credentials issue) #2777

Can't use `optFn` to set region for S3 Express operations (credentials issue) #2777

stefansundin commented Sep 6, 2024

lucix-aws commented Sep 6, 2024 •

edited

Loading

Can't use optFn to set region for S3 Express operations (credentials issue) #2777

Can't use optFn to set region for S3 Express operations (credentials issue) #2777

Comments

stefansundin commented Sep 6, 2024

Acknowledgements

Describe the bug

Expected Behavior

Current Behavior

Reproduction Steps

Possible Solution

Additional Information/Context

AWS Go SDK V2 Module Versions Used

Compiler and Version used

Operating System and version

lucix-aws commented Sep 6, 2024 • edited Loading

Can't use `optFn` to set region for S3 Express operations (credentials issue) #2777

Can't use `optFn` to set region for S3 Express operations (credentials issue) #2777

lucix-aws commented Sep 6, 2024 •

edited

Loading