How to ingest Amazon ElastiCache for Redis Logs via Amazon S3?

[Paul-AWS]

By default, Centralized Logging with OpenSearch don't support ElastiCache for Redis logs in service logs pipelines. How to ingest it via S3 source feature?

[Paul-AWS]

Step 0. Specify ElastiCache for Redis logs delivery into S3 bucket

Please refer to the document below to complete the logs configuration.

Document: Specifying log delivery using the Console

Choose Kinesis Firehose in Destination Type, to make sure delivering logs into S3 bucket.

Step 1. Create a Log Config

1. Go to CLO Console, choose Log Configs on the left navigation bar.

2. Choose Create a log config, type in the Config Name, such as "redis-engine".

3. Choose the log type as "JSON".

4. Type in the sample log in Sample Log text area. You may use the following logs as an example.

{ "CacheClusterId": "test-cluster-0003-001","CacheNodeId": "0002","LogLevel": "NOTICE","Role": "S","Time": "22/Jun/2023:08:30:18","Message": "Successful partial resynchronization with primary." }

5. Choose Parse Log.

6. Change the type of CacheClusterId, CacheNodeId, LogLevel, Role to keyword, so we can use this field to group to build dashboard later.

7. Change the type of Time to date, and input the following in the Time format.

%d/%b/%Y:%H:%M:%S

8. In the Time Key dropdown list, choose "Time".

9. In the Timezone dropdown list, choose "UTC+00:00".

10. Click the button Create.

Step 2. Create a log ingestion pipeline

1. Go to CLO Console, choose Application Log under Log Analytics Pipelines.

2. Choose Create a pipeline, select Amazon S3 as your log source, then click Next.

3. Specify source settings and select/type in the following parameters, and choose Next.

   • Amazon S3 Bucket: "The S3 bucket name that your ElastiCache logs stored in"
   • File path prefix filter: "The prefix name that your ElastiCache logs stored in"
   • Ingestion mode: "On-going"
   • Compression format: "Gzip"

4. On the Log config step, choose redis-engine, and choose Next.

5. On the AOS step, input the index name, such as "redis-engine", and choose the workshop-os as the AOS domain. Remain other parameters as default, and select Next.

6. On the Alarms and tags step, select Create.

You will see the pipeline is in Creating status, and it will take approximate 5~8 minutes to turn into Active.

Step 3. Import a pre-defined Redis Engine log Dashboard

1. Go to the Amazon OpenSearch Dashboards in your browser.

2. Choose Stack Management on the left navigation bar.

3. Select Saved Objects and Import.

4. Download the .ndjson file from the link below and upload it to the drop box.

redis-manual.ndjson

Please select Create new objects with random IDs. Then click Import, an OpenSearch Dashboard and index pattern will automatically be generated for your redis engine index, which allows you to view redis engine log dashboard.

5. If all the steps are correct, it should look like this:

6. Choose Done.

Step 4. View Redis Engine Log Dashboard

1. Go to Dashboards, and choose Redis Engine Dashboard Manual:

2. You can explore the Redis engine now.