[Metricbeat] ActiveMQ module

[mtojek]

Here is the tracking issue for the ActiveMQ metricbeat module development.

Introduction

show/hide

ActiveMQ is a Java-based messaging server. Client applications can be integrated using the following protocols: AQMP, STOMP, MQTT and JMS.

Starting with ActiveMQ 5.8.0 Jolokia is bundled which provides a RESTful interface to ActiveMQ’s JMX capabilities.

(source: https://activemq.apache.org/how-can-i-monitor-activemq)

The module should collect metrics for the broker, all queues and topics.

Metricbeat Module / Dataset release checklist

This checklist is intended for Devs which create or update a module to make sure modules are consistent.

Modules

For a metricset to go GA, the following criterias should be met:

• Supported versions are documented
• Supported operating systems are documented (if applicable)
• Integration tests exist
• System tests exist
• Automated checks that all fields are documented
• Documentation
• Fields follow ECS and naming conventions
• Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)
  • Open issue in EUI repo to add icon for module if not already exists.
• Open PR against Kibana repo with tutorial. Examples can be found here.
• Script in configuration is moved to manifest.yml Move scripts in light module configurations to manifest.yml #14742

Metricbeat module

• Example data.json exists and an automated way to generate it exists (go test -data)
• Test environment in Docker exist for integration system tests

Metrics

This is the list of sample metrics that the server should expose - consider them as basic requirements.

show/hide

Broker

• MemoryPercentUsage - memory percent usage
• StorePercentUsage - store percent usage
• TempPercentUsage - temporary percent usage
• TotalConnectionsCount - total connections count
• TotalConsumerCount - total consumer count
• TotalDequeueCount - total dequeue count
• TotalEnqueueCount - total enqueue count
• TotalMessageCount - total message count
• TotalProducerCount - total producer count

Queue

• AverageEnqueueTime - average enqueue time
• AverageMessageSize - average message size
• ConsumerCount - consumer count
• DequeueCount - dequeue count
• DispatchCount - dispatch count
• EnqueueCount - enqueue count
• ExpiredCount - xxpired count
• InFlightCount - in-flight count
• MaxEnqueueTime - max enqueue Time
• MemoryPercentUsage - memory percent usage
• MinEnqueueTime - min enqueue Time
• ProducerCount - producer count
• QueueSize - queue size

Topic

• AverageEnqueueTime - average enqueue time
• AverageMessageSize - average message size
• ConsumerCount - consumer count
• DequeueCount - dequeue count
• DispatchCount - dispatch count
• EnqueueCount - enqueue count
• ExpiredCount - xxpired count
• InFlightCount - in-flight count
• MaxEnqueueTime - max enqueue Time
• MemoryPercentUsage - memory percent usage
• MinEnqueueTime - min enqueue Time
• ProducerCount - producer count

[mtojek]

FYI @exekias

[exekias]

Great to know that Jolokia is now bundled! that makes using our jolokia metricset even easier. Have a look to tomcat module for an example on how to leverage it here :)

[mtojek]

Agree ;) I believe this will be the first supported version then.

btw. Thanks for adding tags!

[mtojek]

The PR with module implementation has been merged. Also, the checklist above has been updated.

[mtojek]

I noticed an issue with missing Python dependency in Windows environments. After a short investigation, I've raised a GH issue to the Infra team.

[exekias]

ey @mtojek I just saw your issue to get the deps. We don't really run integration tests on windows, due to the requirement limitations, among other things. Your test would actually be skiped by

@unittest.skipUnless(metricbeat.INTEGRATION_TESTS, "integration test")

The problem is that stomp is imported before that. What we normally do is importing extra libraries inside called methods, and not globally. See https://github.com/elastic/beats/blob/master/metricbeat/tests/system/test_kafka.py#L50 as an example.

I think you can do this change and close the infra issue if you want

[mtojek]

Didn't know about that, will fix this in sec. Thanks!

[mtojek]

Due to a different scale of Memory Usage related fields (pct: 1..100 vs 0..1), there is a dedicated PR #14711 (review) to convert the value to the appropriate range.

[mtojek]

The PR with dashboards has been merged. I'm proceeding with updating with Kibana updates (home, icon, etc.)

[mtojek]

The PR with Kibana tutorial has been merged. Backport will be delivered as soon as the branch 7.x in elastic/beats is merged.

[mtojek]

Metricbeat module development has been finished. Remaining work (processor definitions) will be adressed in #14742 .

Resolving

[ChrsMark]

Regression found while testing with BC1:

Error creating runner from config: 1 error: the processor action script does not exist. Valid actions: decode_json_fields, rename, community_id, include_fields, truncate_fields, add_locale, add_observer_metadata, dns, copy_fields, decode_base64_field, drop_fields, extract_array, add_docker_metadata, add_kubernetes_metadata, dissect, add_tags, decompress_gzip_field, add_cloud_metadata, add_host_metadata, add_process_metadata, convert, registered_domain, add_fields, add_labels, drop_event

The script processor usage by this module introduced at #14655 was not included in 7.6 branch after #14711 was merged.
We need to backport #14711 to 7.6 so as the module to be functional.

cc: @mtojek @exekias

Update:
Backports:
#15632
#15633

[ChrsMark]

Tested with BC2 and works as expected 👌 !