Cannot only provide auth.region without setting additional auth config

[bkaznowski]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

When specifying the s3 bucket region, this also overrides the sts endpoint to the same region as the bucket. For example, if the region is set to eu-west-1 then Vector will use sts.eu-west-1.amazonaws.com as the sts endpoint. The sts region can be overwritten by setting auth.region. However, this cannot be set without providing other fields and is contrary to what the documentation says. This is because it won't match any of the AwsAuthentication enum implementations and leads to the following error:

2023-05-16T12:52:30.805806Z ERROR vector::cli: Configuration error. error=data did not match any variant of untagged enum AwsAuthentication
in `sinks.my_bucket`

We run Vector in EKS and so we have an IAM role associated with the Kubernetes service account. However, we have a firewall that restricts egress traffic. This means that we don't have to provide any other auth config for Vector to be able to work but it has to be able to communicate with a whitelisted sts endpoint. Ideally, we don't want to whitelist the regional sts endpoints and would like to either have a way to force it to either use the global endpoint or for Vector to allow overriding the sts region without having to provide additional auth fields.

Configuration

[sources.my_source_id]
type = "file"
include = [ "/var/log/**/*.log" ]
[sinks.my_bucket]
type = "aws_s3"
inputs = [ "my_source_id" ]
bucket = "my-logs"
region = "eu-west-1"
auth.region = "eu-west-2"

Version

vector 0.29.1 (x86_64-unknown-linux-musl 74ae15e 2023-04-20 14:50:42.739094536)

Debug Output

2023-05-16T13:08:58.144166Z DEBUG vector::app: Internal log rate limit configured. internal_log_rate_secs=10
2023-05-16T13:08:58.144202Z  INFO vector::app: Log level is enabled. level="vector=trace,codec=trace,vrl=trace,file_source=trace,tower_limit=trace,rdkafka=trace,buffers=trace,lapin=trace,kube=trace"
2023-05-16T13:08:58.144536Z  INFO vector::app: Loading configs. paths=["/etc/vector/vector.toml"]
2023-05-16T13:08:58.144867Z DEBUG vector::config::loading: No secret placeholder found, skipping secret resolution.
2023-05-16T13:08:58.145083Z ERROR vector::cli: Configuration error. error=data did not match any variant of untagged enum AwsAuthentication
in `sinks.my_bucket`

Example Data

No response

Additional Context

• Vector is running in Kubernetes
• The S3 bucket is in eu-west-1
• The cluster is in eu-west-2
• There is a firewall that prevents egress. So only the global and eu-west-2 sts endpoints are whitelisted

References

No response

[jszwedko]

Hi @bkaznowski !

Thanks for filing this. Can you can confirm the authentication mechanisms you are expecting to use? You mention STS which would imply assume-role, but I'm not seeing a role specified here. Are the credentials originally supposed to be fetched from IMDSv2?

[andrewnicolalde]

Hi @jszwedko, I don't think we're using IMDS2. We're running Vector in a pod on EKS, and making use of IAM Roles for Service Accounts (IRSA) to provide credentials to Vector.

As I understand it, IRSA provides temporary credentials to workloads when they make an AssumeRoleWithWebIdentity call.

More detailed info on how IRSA works: https://aws.amazon.com/blogs/containers/diving-into-iam-roles-for-service-accounts/

[jszwedko]

Gotcha, thanks @andrewnicolalde . Did you try not using auth.region at all? Did you encounter an error when you did?

[bkaznowski]

Yes, we did try. Without auth.region we were getting a timeout on the healthcheck due to the firewall and with auth.region we were getting the error outlined in the description where it can't map to the AwsAuthentication enum

[jszwedko]

Thanks! I think I understand the issue. I opened #17414 to address it.

[bkaznowski]

Excellent, thanks! That looks like it should fix the issue