x-pack/filebeat/input/awss3: allow cross-region bucket configuration #40309
Conversation
efd6
added
Filebeat
botelastic
bot
added
needs_team
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
| opts := a.client.Options() | ||
| if opts.Region == region { | ||
| return a.client | ||
| } | ||
| opts.Region = region | ||
| return s3.New(opts) |
There was a problem hiding this comment.
Depending on the frequency of the new-client path, this could generate a reasonable amount of work. Something that I considered but did not add is having awsS3API hold a map[string]*s3.Client that can cache (non-retiring) the clients for each region. Given that there are not large numbers of regions, it should be reasonable to retain clients that are constructed dynamically for the life of the awsS3API. WDYT?
There was a problem hiding this comment.
I think this is a good idea. My intuition is that in the use-cases where this is needed that it can be in very high volumes (big orgs, lots of aws projects, lots of regions, all notifications centralized to one queue).
There was a problem hiding this comment.
I've added caching and limited the size to 100. We should never hit this limit, but I wanted to make sure that we don't put ourselves in the situation that this results in an OoM crash.
|
This pull request is now in conflicts. Could you fix it? 🙏 |
andrewkroh
added
bug
bugfix
backport-8.15
|
I added the backport label thinking that this can try to target v8.15.1. |
efd6
removed
the
backport-skip
|
@efd6 Should we create a separate PR to update the documentation too? Thanks! |
|
@kaiyan-sheng I had not intended to include documentation since this feature needs no user understanding. |
When a region is not specified in the SQS notification then reuse the existing S3 client instead of creating a new one based on an empty (unspecified) AWS region name. This problem affected custom SQS notification formats that did not specify a region name (like Crowdstrike FDR notifications). Fixes: elastic/integrations/elastic#10647 Relates: elastic#40309
When a region is not specified in the SQS notification then reuse the existing
S3 client instead of creating a new one based on an empty (unspecified) AWS
region name. This problem affected custom SQS notification formats that did
not specify a region name (like Crowdstrike FDR notifications).
This addresses errors like:
S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name.
Fixes: elastic/integrations/elastic#10647
Relates: elastic#40309
…#40628) When a region is not specified in the SQS notification then reuse the existing S3 client instead of creating a new one based on an empty (unspecified) AWS region name. This problem affected custom SQS notification formats that did not specify a region name (like Crowdstrike FDR notifications). This addresses errors like: S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name. Fixes: elastic/integrations/#10647 Relates: #40309
…#40628) When a region is not specified in the SQS notification then reuse the existing S3 client instead of creating a new one based on an empty (unspecified) AWS region name. This problem affected custom SQS notification formats that did not specify a region name (like Crowdstrike FDR notifications). This addresses errors like: S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name. Fixes: elastic/integrations/#10647 Relates: #40309 (cherry picked from commit 6d25d46)
…#40628) (#40632) When a region is not specified in the SQS notification then reuse the existing S3 client instead of creating a new one based on an empty (unspecified) AWS region name. This problem affected custom SQS notification formats that did not specify a region name (like Crowdstrike FDR notifications). This addresses errors like: S3 download failure: s3 GetObject failed: operation error S3: GetObject, resolve auth scheme: resolve endpoint: endpoint rule error, Invalid region: region was not a valid DNS name. Fixes: elastic/integrations/#10647 Relates: #40309 (cherry picked from commit 6d25d46) Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
Proposed commit message
See title.
Checklist
CHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.Disruptive User Impact
Author's Checklist
How to test this PR locally
Related issues
Use cases
Screenshots
Logs