Filebeat S3 input support for AWS WAF logs (support application/octet-stream)

[rubal033]

Filebeat S3 input support for AWS WAF logs (support application/octet-stream)

As of now, there is Filebeat s3 input that doesn't support AWS WAF logs. AWS WAF logs use Kinesis Firehose to get to S3 and the "Content-type" is set to "application/octet-stream". Due to this, the logs didn't get expanded in the Elasticsearch.

If that support can be added it will resolve a big problem and many users looking forward to getting AWS WAF logs to the ELK stack for the analysis.

Thanks in Advance

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[legoguy1000]

Can you provide more details? Are u saying the WAF logs aren't plain text files? Can you provide sample logs? Looking at this, https://www.wafcharm.com/en/blog/aws-waf-full-log-s3-output/, it should be json logs in the s3 bucket?

[rubal033]

AWS WAF logs are in JSON format but when Kinesis Firehose put them in s3 bucket it set the metadata "content-type" to "application/octet-stream" instead of "application/json".

So when filebeat s3 plugin read that data it gets its content type as "application/octet-stream" and not able to expand to get the JSON field read properly in Elasticsearch. The whole JSON log get inside "message" field.

[andrewkroh]

The aws-s3 input allows you to override the Content-Type by setting content_type. So if you set content_type: application/json then the content will be parsed as JSON.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html#input-aws-s3-content_type

[andrewkroh]

I think we can close this given we have a solution for the Content-Type. There's a separate request in #28121 to add AWS WAF to the Filebeat AWS module.

[andrewkroh]

Fixed by #25772

[rubal033]

Thank you !

That is really helpful. I will try it and post my observations.