Add support of a new RabbitMQ timestamp format

[leweafan]

Describe the enhancement:

Add RabbitMQ timestamp format with nanosecond precision and timezone "2022-03-23 13:16:58.000369+03:00".
Latest version switched to new default timestamp format and it's not supported by filebeat rabbitmq module.
Previous timestamp format "2019-04-12 10:11:15.094".

Now RabbitMQ ingest pipeline using DATETIME pattern and it should be replaced by TIMESTAMP_ISO8601 (see these comments about this).

RabbitMQ logging description - https://www.rabbitmq.com/logging.html

Describe a specific use case for the enhancement or feature:

2022-03-23 13:16:58.000369+03:00 [warning] <0.24873.39> client unexpectedly closed TCP connection

This issue applies to the package integration and to the Filebeat module.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Ping @elastic/obs-service-integrations.

[jsoriano]

Updated description with reference to these comments elastic/beats#13879 (comment), and moved to the integrations repository as this applies to the package integration and to the Filebeat module.

[JanKnipp]

I'm seeing the same issue with the agent integration for rabbitmq (which of course uses filebeat)

2022-11-24 07:37:06.753000+01:00 [error] <0.30307.23> vhost xxxx not found

this basically makes the log integration for rabbitmq "useless" as grok pattern matching will always fail.

the change is not too hard

[ { "set": { "field": "event.ingested", "value": "{{_ingest.timestamp}}" } }, { "set": { "field": "ecs.version", "value": "8.0.0" } }, { "set": { "field": "event.kind", "value": "event" } }, { "set": { "field": "event.type", "value": "info" } }, { "rename": { "field": "message", "target_field": "event.original", "ignore_missing": true } }, { "grok": { "field": "event.original", "patterns": [ "%{TIMESTAMP_ISO8601:timestamp} \\[%{WORD:log.level}\\] %{ERL_PID:rabbitmq.log.pid} %{GREEDYMULTILINE:message}" ], "pattern_definitions": { "GREEDYMULTILINE": "(.|\n)*", "ERL_PID": "\\<%{INT}+\\.%{INT}+\\.%{INT}+\\>" }, "ignore_missing": true } }, { "date": { "field": "timestamp", "formats": [ "yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ" ], "target_field": "@timestamp", "if": "ctx.event.timezone == null" } }, { "date": { "field": "timestamp", "formats": [ "yyyy-MM-dd HH:mm:ss.SSSSSSZZZZZ" ], "target_field": "@timestamp", "timezone": "{{ event.timezone }}", "if": "ctx.event.timezone != null" } }, { "remove": { "field": [ "timestamp" ] } }, { "remove": { "field": "event.original", "if": "ctx?.tags == null || !(ctx.tags.contains('preserve_original_event'))", "ignore_failure": true, "ignore_missing": true } }, { "pipeline": { "name": "logs-rabbitmq.log@custom", "ignore_missing_pipeline": true } } ]

[ritalwar]

The change has been made to support new timestamp format.
Tested and is working fine.
PR link for integrations: #4918
PR link for filebeat: elastic/beats#34211

[botelastic]

Hi! We just realized that we haven't looked into this issue in a while. We're sorry! We're labeling this issue as Stale to make it hit our filters and make sure we get back to it as soon as possible. In the meantime, it'd be extremely helpful if you could take a look at it as well and confirm its relevance. A simple comment with a nice emoji will be enough :+1. Thank you for your contribution!