How to ingest ElastiCache for Redis Slow Log into OpenSearch (CLO) v2.0.0

[owenCCY]

For the Centralized Logging with OpenSearch Solution version v2.0.0, ElastiCache for Redis Slow Log is NOT supported in service log ingestion. This guidance will help you go through the ingestion steps.

[owenCCY]

Step1. Create log config for ElastiCache Redis slow log in CLO console

You can paste the following slow log example:

{
    "CacheClusterId": "logslowxxxxmsxj",
    "CacheNodeId": "0001",
    "Id": 296,
    "Timestamp": 1605631822,
    "Duration (us)": 0,
    "Command": "GET ... (1 more arguments)",
    "ClientAddress": "192.168.12.104:55452",
    "ClientName": "logslowxxxxmsxj##"
}

[owenCCY]

Step2. Create ingestion pipeline for ElastiCache Redis Slow Logs in CLO console

Go to application log section in CLO, create an application pipeline which takes Amazon S3 as log source

Create source settings, the S3 bucket and Prefix you choose should be the place you planned for your ElastiCache logging location

[owenCCY]

Step3. Change index templates

Go to Dev Tools in OpenSearch dashboard:

Use the following command to view existing indices:

GET _cat/indices?v

Please find out the index you created for ElastiCache Redis Slow Log, and copy the name of the index prefix.

Let's say you name it as example-elasticache-for-redis. Please substitute with the name of your own index name!

Delete the existing example-elasticache-for-redis index using the following command:

DELETE example-elasticache-for-redis*

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

Get the index template of the existing index template using the following command:

GET _index_template/example-elasticache-for-redis-template

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

Please copy the returned index template and modify the value of Timestamp field with the following value:

"Timestamp" : {
  "format" : "epoch_millis",
  "type" : "date"
}

Update the index template with the modified index template, the command should look like the following:

PUT _index_template/example-elasticache-for-redis-template
{
  "index_patterns" : [
    "example-elasticache-for-redis-*"
  ],
  "template" : {
    "settings" : {
      "index" : {
        "codec" : "best_compression",
        "refresh_interval" : "1s",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "plugins" : {
          "index_state_management" : {
            "rollover_alias" : "example-elasticache-for-redis"
          }
        }
      }
    },
    "mappings" : {
      "properties" : {
        "ClientName" : {
          "type" : "text"
        },
        "CacheClusterId" : {
          "type" : "text"
        },
        "CacheNodeId" : {
          "type" : "text"
        },
        "Command" : {
          "type" : "text"
        },
        "Duration+%28us%29" : {
          "type" : "text"
        },
        "ClientAddress" : {
          "type" : "text"
        },
        "Id" : {
          "type" : "integer"
        },
        "Timestamp" : {
          "format" : "epoch_second",
          "type" : "date"
        }
      }
    }
  },
  "composed_of" : [ ]
}

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

Create the alias mapping for example-elasticache-for-redis index

PUT %3Cexample-elasticache-for-redis-%7Bnow%7Byyyy-MM-dd%7D%7D-000001%3E
{
  "aliases": {
    "example-elasticache-for-redis": {
      "is_write_index": true
    }
  }
}

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

[owenCCY]

Step4. Enable ElastiCache Redis Slow Logs in your cluster settings

Go to ElastiCache > Redis Cluster in AWS console and enable slow logs

Select JSON as your Log Format, select Kinesis Data Firehose as your log destination type.

Choose or create a log delivery stream which take Amazon S3 as destination, this S3 location should be the previously selected location during application pipeline creation process:

[owenCCY]

Step5. create index pattern for your ElastiCache Redis index

Go to the following location in OpenSearch dashboard and create index:

Type in the index name and click Next Step

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

Select Timestamp as your Time Field and click Create Index Pattern:

Note: you need to substitute the index name example-elasticache-for-redis with the one you created.

[owenCCY]

Step5. View logs in OpenSearch

Please go to Discover page in OpenSearch and select the correct index name:

You can now see the details of your ElastiCache Redis Slow Logs.

[luorobin-a2z]

can we have some dashboard for redis slow query log? Just some a few as below

• Number of slow logs per time - Bar chart with time serie
• Number of slow logs per CacheNodeId - Pipe chart
• Max durations, average duration, minimum duration over time - line chart time series
• Top Key command by slow query number or slow query duration - table and bar chart