[filebeat] Credentials error with S3+SQS input in 8.4.0

[tommyers-elastic]

To repro, setup an AWS logs integration with elastic agent running in EC2 and the S3+SQS input. You will likely see SQS errors like this:

sqs ReceiveMessage failed: operation error SQS: ReceiveMessage, https response error StatusCode: 403, RequestID: cb57783a-505f-5099-9160-23b8eea8ddbb, api error SignatureDoesNotMatch: Credential should be scoped to a valid region. 

We believe this is due to changes in the SDK brought in with #31224. It looks like the SQS endpoint SigningRegion is not being set in the SDK configuration.

The reason for this appears to be that we are using a custom endpoint resolver when building the S3/SQS configuration. At this point it's unclear to me why we aren't just using the default resolver here. I can't see what benefits we get by maintaining the custom version.

[aspacca]

I was not able to reproduce from awss3 integration test with filebeat running on a local machine, providing only AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and renaming ~/.aws folder so that the sdk won't be able to find it

I wonder if this is specific to running filebeat on an ec2 instance and having the sdk relying on the ec2 instance metadata

[tommyers-elastic]

we have discovered a workaround for this for now, which is to remove the custom endpoint configuration from either the fleet managed or filebeat configuration. the value must be removed entirely, not left as the default value of amazonaws.com.

it's still not 100% clear why this affects the credential scope, but we are continuing to work on figuring it out.

we believe the reason that this is not immediately reproducible with standalone filebeat compared to agent is that OOTb, the filebeat configuration does not include the custom endpoint config, whereas in agent, it defaults to amazonaws.com

[ThomasLarge]

we have discovered a workaround for this for now, which is to remove the custom endpoint configuration from either the fleet managed or filebeat configuration. the value must be removed entirely, not left as the default value of amazonaws.com.

it's still not 100% clear why this affects the credential scope, but we are continuing to work on figuring it out.

we believe the reason that this is not immediately reproducible with standalone filebeat compared to agent is that OOTb, the filebeat configuration does not include the custom endpoint config, whereas in agent, it defaults to amazonaws.com

I can confirm this has worked for me too, not sure what has happened but we didn't get this error before updating

[andrewkroh]

@elastic/obs-cloud-monitoring could you update this issue with some more details including a list of affected versions and what version contains the fix?

Do any Fleet integrations that use aws-s3 required updates or will they be fixed automatically with the fixed version of Agent/Beats? The list of Fleet integrations referencing endpoint with aws-s3 is:

• aws
• aws_logs
• cisco_umbrella
• cloudflare_logpush
• crowdstrike

Known issue documentation: https://www.elastic.co/guide/en/beats/libbeat/8.4/release-notes-8.4.1.html#_known_issue.

[kaiyan-sheng]

@aspacca Please correct me if I'm wrong.

Affected Beats version: 8.4.0 and 8.4.1
Beats version that contains the fix: 8.4.2 and above

When user is not using the custom endpoint configuration, they can upgrade the following packages as a workaround:
aws 1.23.4
aws_logs 0.2.5
cisco_umbrella 1.3.1
cloudflare_logpush (sorry we missed this one, I just created a PR elastic/integrations#4207 for it)
crowdstrike 1.5.1

Once 8.4.2 Beats version is released, my understanding is this should be fixed automatically.

[aspacca]

@andrewkroh
a little more context on the details:
with the upgrade of the aws sdk in #31224 our custom endpoint resolver was broken, since missing some information required by the new version of the sdk for signing the requests

our custom endpoint resolver was set only when endpoint is not empty: with an empty endpoint the internal sdk resolvers will be used for every service.

1. setting a default empty endpoint prevented the bug to be hit
2. in the proper fix at Refactor endpoint resolver for AWS services #32921 we just removed at all using a custom endpoint resolver but for setting some specific endpoint resolver options for fips (in general) and using a custom full URL endpoint (only in the case of non aws s3 compatible bucket in the direct filebeat s3 input)
3. we have a PR from @legoguy1000 at [AWS] Expose default region config integrations#4158 to remove at all exposing the endpoint setting and indeed expose the default_region one: this last is needed in case no aws region is set anywhere else (env variable, credentials or instance profile) and we are in a non-standard aws cloud (gov, cn, etc)

[legoguy1000]

I'll have to update my PR with cloudflare logpush as that hadn't been merged when I wrote it.

[aspacca]

@legoguy1000 , @kaiyan-sheng is already taking care of it with elastic/integrations#4207

thanks :)