Cherry-pick #15917 to 7.6: [Metricbeat] Add dedot for cloudwatch metric name

[kaiyan-sheng]

Cherry-pick of PR #15917 to 7.6 branch. Original message:

What does this PR do?

This PR is to add automatic dedot for metircs from Cloudwatch.

Why is it important?

Without this PR, when cloudwatch metricset trying to collect from namespace AWS/Firehose, user would get an error about failed to parse field:

"reason":"failed to parse field [aws.firehose.metrics.PutRecordBatch.Records] of type [double] in document with id 'KQC47W8BlbxP6epG4o0l'. Preview of field's value: '{avg=1}'"

Here is the full error message:

2020-01-28T19:37:07.710Z	WARN	elasticsearch/client.go:517	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf844130ce47b20d, ext:3698387180, loc:(*time.Location)(0x7eb1ba0)}, Meta:null, Fields:{"agent":{"ephemeral_id":"f9321cda-c194-4250-981d-897638e99888","hostname":"stack-7-6-0-test","id":"74fe85f9-65e1-4ac3-8561-e5450bc4e923","type":"metricbeat","version":"7.6.0"},"aws":{"cloudwatch":{"namespace":"AWS/Firehose"},"dimensions":{"DeliveryStreamName":"phone-home-staging-xpack"},"firehose":{"metrics":{"BytesPerSecondLimit":{"avg":5242880.000000},"DeliveryToS3":{"Bytes":{"avg":11465.000000},"DataFreshness":{"avg":61.000000},"Records":{"avg":1.000000},"Success":{"avg":1.000000}},"IncomingBytes":{"avg":11465.000000},"IncomingPutRequests":{"avg":1.000000},"IncomingRecords":{"avg":1.000000},"KMSKeyAccessDenied":{"avg":0.000000},"KMSKeyDisabled":{"avg":0.000000},"KMSKeyInvalidState":{"avg":0.000000},"KMSKeyNotFound":{"avg":0.000000},"PutRecordBatch":{"Bytes":{"avg":11465.000000},"Latency":{"avg":9.000000},"Records":{"avg":1.000000},"Requests":{"avg":1.000000}},"PutRequestsPerSecondLimit":{"avg":2000.000000},"RecordsPerSecondLimit":{"avg":5000.000000},"ThrottledRecords":{"avg":0.000000}}}},"cloud":{"account":{"id":"627959692251","name":"elastic-test"},"provider":"aws","region":"us-west-2"},"ecs":{"version":"1.4.0"},"event":{"dataset":"aws.cloudwatch","duration":28562828950,"module":"aws"},"host":{"architecture":"x86_64","containerized":false,"hostname":"stack-7-6-0-test","id":"45661bd1e51648e38783c568c541c5ac","name":"stack-7-6-0-test","os":{"codename":"bionic","family":"debian","kernel":"4.15.0-66-generic","name":"Ubuntu","platform":"ubuntu","version":"18.04.3 LTS (Bionic Beaver)"}},"metricset":{"name":"cloudwatch","period":300000},"service":{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.firehose.metrics.DeliveryToS3.DataFreshness] of type [double] in document with id '2zao7W8BuJq_FrJ2QkUq'. Preview of field's value: '{avg=61}'","caused_by":{"type":"json_parse_exception","reason":"Current token (START_OBJECT) not numeric, can not use numeric value accessors\n at [Source: org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper@72aee4fc; line: 1, column: 860]"}}

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

Enable aws module in Metricbeat and change aws.yml config file to use the one below:

- module: aws
  period: 5m
  metricsets:
    - cloudwatch
  credential_profile_name: elastic-test
  metrics:
    - namespace: AWS/Firehose

You should see metrics from AWS/Firehose in Elasticsearch with metric names like:
aws.firehose.metrics.PutRecordBatch_Bytes.avg
aws.firehose.metrics.PutRecordBatch_Latency.avg
aws.firehose.metrics.PutRecordBatch_Records.avg

Related issues

• Closes Fail to index AWS events with fields with one more level in their name paths #15916

Screenshots

Here is an event after dedot cloudwatch metric name:

{
  "_index": "metricbeat-8.0.0-2020.01.27-000001",
  "_type": "_doc",
  "_id": "LYnq7W8BZy5rQOjN2Rry",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-01-28T20:49:33.370Z",
    "ecs": {
      "version": "1.4.0"
    },
    "aws": {
      "firehose": {
        "metrics": {
          "PutRecordBatch_Records": {
            "max": 1,
            "min": 1,
            "sum": 1,
            "count": 1,
            "avg": 1
          },
          "KMSKeyInvalidState": {
            "sum": 0,
            "count": 286,
            "avg": 0,
            "max": 0,
            "min": 0
          },
          "PutRecordBatch_Bytes": {
            "min": 9947,
            "sum": 9947,
            "count": 1,
            "avg": 9947,
            "max": 9947
          },
          "BytesPerSecondLimit": {
            "avg": 5242880,
            "max": 5242880,
            "min": 5242880,
            "sum": 5242880,
            "count": 1
          },
          "KMSKeyNotFound": {
            "sum": 0,
            "count": 286,
            "avg": 0,
            "max": 0,
            "min": 0
          },
          "ThrottledRecords": {
            "sum": 0,
            "count": 1,
            "avg": 0,
            "max": 0,
            "min": 0
          },
          "IncomingBytes": {
            "count": 1,
            "avg": 9947,
            "max": 9947,
            "min": 9947,
            "sum": 9947
          },
          "IncomingRecords": {
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 1,
            "count": 1
          },
          "KMSKeyAccessDenied": {
            "max": 0,
            "min": 0,
            "sum": 0,
            "count": 286,
            "avg": 0
          },
          "PutRecordBatch_Requests": {
            "count": 1,
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 1
          },
          "RecordsPerSecondLimit": {
            "avg": 5000,
            "max": 5000,
            "min": 5000,
            "sum": 5000,
            "count": 1
          },
          "KMSKeyDisabled": {
            "max": 0,
            "min": 0,
            "sum": 0,
            "count": 286,
            "avg": 0
          },
          "IncomingPutRequests": {
            "count": 1,
            "avg": 1,
            "max": 1,
            "min": 1,
            "sum": 1
          },
          "PutRecordBatch_Latency": {
            "count": 1,
            "avg": 11,
            "max": 11,
            "min": 11,
            "sum": 11
          },
          "PutRequestsPerSecondLimit": {
            "avg": 2000,
            "max": 2000,
            "min": 2000,
            "sum": 2000,
            "count": 1
          }
        }
      },
      "cloudwatch": {
        "namespace": "AWS/Firehose"
      },
      "dimensions": {
        "DeliveryStreamName": "phone-home-staging-xpack"
      }
    },
    "event": {
      "dataset": "aws.cloudwatch",
      "module": "aws",
      "duration": 14612705477
    },
    "metricset": {
      "name": "cloudwatch",
      "period": 300000
    },
    "service": {
      "type": "aws"
    },
    "cloud": {
      "provider": "aws",
      "region": "us-west-2",
      "account": {
        "name": "elastic-test",
        "id": "627959692251"
      }
    },
    "host": {
      "name": "KaiyanMacBookPro",
      "id": "9C7FAB7B-29D1-5926-8E84-158A9CA3E25D",
      "hostname": "KaiyanMacBookPro",
      "architecture": "x86_64",
      "os": {
        "version": "10.13.6",
        "family": "darwin",
        "name": "Mac OS X",
        "kernel": "17.7.0",
        "build": "17G10021",
        "platform": "darwin"
      }
    },
    "agent": {
      "type": "metricbeat",
      "ephemeral_id": "5a43913b-3cb7-4c85-8963-4319728554a8",
      "hostname": "KaiyanMacBookPro",
      "id": "23fd6ee9-2255-43f3-9565-78a051337c04",
      "version": "8.0.0"
    }
  },
  "fields": {
    "@timestamp": [
      "2020-01-28T20:49:33.370Z"
    ]
  },
  "sort": [
    1580244573370
  ]
}