Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filebeat skips ingesting CloudTrail logs that aren't in S3 bucket root #23109

Closed
borrell opened this issue Dec 14, 2020 · 2 comments · Fixed by #23126
Closed

Filebeat skips ingesting CloudTrail logs that aren't in S3 bucket root #23109

borrell opened this issue Dec 14, 2020 · 2 comments · Fixed by #23126
Assignees
@leehinman
Labels
Team:Platforms Label for the Integrations - Platforms team

Comments

@borrell
Copy link

borrell commented Dec 14, 2020 •
edited
Loading

Since #21088 was included, Filebeat will ignore CloudTrail logs when they are not in the root of the S3 bucket path. This situation can occur if the user is using AWS Organizations to manage their CloudTrail logs, or if they have manually set an S3 Prefix in their CloudTrail configuration.

The following configuration included with the module is intended to be used to filter out different log types based on the user configuration. If a filter is set to allow a log type (e.g. CloudTrail-Digest logs), then the regex is parsed. If the input matches the regex, the logfiles are further expanded and ingested. If a logfile does not match the regex, it is discarded/ignored.

beats/x-pack/filebeat/module/aws/cloudtrail/config/s3.yml

Line 5 in 49c8d87

- regex: '^AWSLogs/\d+/CloudTrail/'

beats/x-pack/filebeat/module/aws/cloudtrail/config/s3.yml

Line 10 in 49c8d87

- regex: '^AWSLogs/\d+/CloudTrail-Digest/'

beats/x-pack/filebeat/module/aws/cloudtrail/config/s3.yml

Line 14 in 49c8d87

- regex: '^AWSLogs/\d+/CloudTrail-Insight/'

This regex accounts for the default AWS CloudTrail log destination, in which a log object will have the key
AWSLogs/1234567890/CloudTrail/....

however does not account for users specifying a prefix manually, or using AWS Organizations

custom-prefix/AWSLogs/1234567890/CloudTrail/.... or
o-xxxxxxx/AWSLogs/1234567890/CloudTrail/....

I have resolved this in testing by modifying the beginning of the regex to match * up until the AWSLogs string - for example
^*./AWSLogs/\d+/CloudTrail/

I searched around and also found this reference to the issue https://discuss.elastic.co/t/request-for-filebeat-aws-cloudtrail-documentation-configuration-options-for-s3prefix-etc/256801

This failure mode will not appear in logs, as the filter simply ignores any logfiles that do not match the specified path, and handles them accordingly - it's not an error, just a use-case that isn't handled as expected
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Dec 14, 2020
@andresrc andresrc added Team:Platforms Label for the Integrations - Platforms team Team:Security-External Integrations labels Dec 14, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Dec 14, 2020
@leehinman leehinman self-assigned this Dec 14, 2020
leehinman added a commit to leehinman/beats that referenced this issue Dec 14, 2020
@leehinman
fix organization and customer prefix for aws/cloudtrail
62930c8 
- use ^.*AWSLogs as beginning of prefix, matches:
  + custom-prefix/AWSLogs/1234567890/CloudTrail/
  + o-xxxxxxx/AWSLogs/1234567890/CloudTrail/
  + AWSLogs/1234567890/CloudTrail/

Closes elastic#23109
@leehinman leehinman mentioned this issue Dec 14, 2020
Merged
3 tasks
leehinman added a commit that referenced this issue Dec 15, 2020
@leehinman @andrewkroh
[Filebeat] fix organization and customer prefix for aws/cloudtrail (#…
b4aba8e 
…23126)

* fix organization and customer prefix for aws/cloudtrail

- use ^.*AWSLogs as beginning of prefix, matches:
  + custom-prefix/AWSLogs/1234567890/CloudTrail/
  + o-xxxxxxx/AWSLogs/1234567890/CloudTrail/
  + AWSLogs/1234567890/CloudTrail/

Closes #23109

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
leehinman added a commit to leehinman/beats that referenced this issue Dec 15, 2020
@leehinman
[Filebeat] fix organization and customer prefix for aws/cloudtrail (e…
aca462c 
…lastic#23126)

* fix organization and customer prefix for aws/cloudtrail

- use ^.*AWSLogs as beginning of prefix, matches:
  + custom-prefix/AWSLogs/1234567890/CloudTrail/
  + o-xxxxxxx/AWSLogs/1234567890/CloudTrail/
  + AWSLogs/1234567890/CloudTrail/

Closes elastic#23109

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit b4aba8e)
@leehinman leehinman mentioned this issue Dec 15, 2020
Merged
3 tasks
leehinman added a commit that referenced this issue Dec 15, 2020
@leehinman
[Filebeat] fix organization and customer prefix for aws/cloudtrail (#…
97d9aad 
…23126) (#23138)

* fix organization and customer prefix for aws/cloudtrail

- use ^.*AWSLogs as beginning of prefix, matches:
  + custom-prefix/AWSLogs/1234567890/CloudTrail/
  + o-xxxxxxx/AWSLogs/1234567890/CloudTrail/
  + AWSLogs/1234567890/CloudTrail/

Closes #23109

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>
(cherry picked from commit b4aba8e)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] fix organization and customer prefix for aws/cloudtrail leehinman/beats
4 participants
@borrell @andresrc @elasticmachine @leehinman