[Metricbeat] Add dedot to aws module when collecting tags #15843
Labels
bug
Metricbeat
Metricbeat
Team:Integrations
Label for the Integrations team
Team:Platforms
Label for the Integrations - Platforms team
In aws module, ec2 metricset collects tags by default. AWS does not limit using dot (
.
) when creating tags for EC2 instances: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_Tags.html#tag-restrictionsIf a tag key includes dot, then ec2 metricset will fail with error like:
2020-01-24T14:44:46.013-0700 WARN elasticsearch/client.go:516 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbf82f72e084b7960, ext:2131556658, loc:(*time.Location)(0x9bcef40)}, Meta:null, Fields:{"agent":{"ephemeral_id":"6374bf9d-4e60-4862-ada1-6b3165a5a81f","hostname":"KaiyanMacBookPro","id":"23fd6ee9-2255-43f3-9565-78a051337c04","type":"metricbeat","version":"8.0.0"},"aws":{"ec2":{"cpu":{"credit_balance":169.591847,"credit_usage":0.004121,"surplus_credit_balance":0.000000,"surplus_credits_charged":0.000000,"total":{"pct":0.099454}},"diskio":{"read":{"bytes":0.000000,"bytes_per_sec":0.000000,"count":0.000000,"count_per_sec":0.000000},"write":{"bytes":0.000000,"bytes_per_sec":0.000000,"count":0.000000,"count_per_sec":0.000000}},"instance":{"core":{"count":1},"image":{"id":"ami-062f7200baf2fa504"},"monitoring":{"state":"disabled"},"private":{"dns_name":"ip-172-31-19-10.ec2.internal","ip":"172.31.19.10"},"public":{"dns_name":"ec2-3-95-172-1.compute-1.amazonaws.com","ip":"3.95.172.1"},"state":{"code":16,"name":"running"},"threads_per_core":1},"network":{"in":{"bytes":530.400000,"bytes_per_sec":1.768000,"packets":6.800000,"packets_per_sec":0.022667},"out":{"bytes":461.600000,"bytes_per_sec":1.538667,"packets":7.400000,"packets_per_sec":0.024667}},"status":{"check_failed":0,"check_failed_instance":0,"check_failed_system":0}},"tags":{"app":{"kubernetes":{"io/name":"foo"}},"created-by":"ks","kubernetes":"{io/cluster/lakeplacid-eksCluster-0a550bd=owned}"}},"cloud":{"account":{"id":"428152502467","name":"elastic-beats"},"availability_zone":"us-east-1c","instance":{"id":"i-00123db57112ab0b6"},"machine":{"type":"t2.micro"},"provider":"aws","region":"us-east-1"},"ecs":{"version":"1.4.0"},"event":{"dataset":"aws.ec2","duration":20819094576,"module":"aws"},"host":{"architecture":"x86_64","hostname":"KaiyanMacBookPro","id":"9C7FAB7B-29D1-5926-8E84-158A9CA3E25D","name":"KaiyanMacBookPro","os":{"build":"17G10021","family":"darwin","kernel":"17.7.0","name":"Mac OS X","platform":"darwin","version":"10.13.6"}},"metricset":{"name":"ec2","period":300000},"service":{"type":"aws"}}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [aws.tags.app] of type [keyword] in document with id '00uD2W8Bcprh94r7uDqM'. Preview of field's value: '{kubernetes={io/name=foo}}'","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:1042"}}
This is also an issue needs to be fixed in cloudwatch metricset.
Discuss forum link: https://discuss.elastic.co/t/configuring-aws-ec2-on-metricbeat-k8-deamonset-throws-errors
The text was updated successfully, but these errors were encountered: