How to create a pipeline to collect logs from multiple resources of the same kind in one index?

[Paul-AWS]

I’d like to ingest logs from multiple resources into one index, to view them all in one dashboard. For example, i have 10 WAF rules, and i would like to ingest all logs from these 10 WAF rules into one index, to view them in one dashboard, without creating 10 pipelines. How to accomplish this?

[Paul-AWS]

By default, one pipeline only ingests logs from one resource into one index in OpenSearch. For several services, Centralized Logging with OpenSearch console provides “Manual” creation method to allow you ingest logs from multiple resources into one index. Here’s the list of supported services.

• S3 access logs
• CloudFront logs (standard)
• Elastic Load Balancer access logs
• WAF logs (full)
• VPC flow logs
• Config logs

Follow the steps below to use “Manual” creation method.

1. Configure multiple resources to write logs into the same S3 location.
2. In the console, choose the “AWS Service Log” in the left sidebar. Then click the “Create log ingestion” button.
3. In “Select an AWS Service”, choose a service in the list, then click “Next”.
4. In “Creation Method”, choose “Manual”, then fill up the resource name and S3 log location parameter, then click “Next”.
5. Set “OpenSearch domain” and “Log Lifecycle” as need, then click “Next”.
6. Add tags if you need, then click “Create” button to create the pipeline.

Then the logs from these resources will be ingested into one single index, and you can view them in on single dashboard. You can also use CloudFormation stack to deploy the pipeline, just fill the S3 log location as the location you write all the logs into.