Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Create ActiveMQ module #14840

Merged
merged 14 commits into from
Dec 2, 2019

Conversation

mtojek
Copy link
Contributor

@mtojek mtojek commented Nov 28, 2019

This PR introduces the Filebeat module for the ActiveMQ platform. It supports two kinds of log entries - standard and audit logs.

$ GENERATE=1 INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=activemq MODULES_PATH=`pwd`/module nosetests tests/system/test_xpack_modules.py

Master issue: #14744

@mtojek mtojek requested a review from a team as a code owner November 28, 2019 11:53
@mtojek mtojek added module in progress Pull request is currently in progress. needs_backport PR is waiting to be backported to other branches. Team:Integrations Label for the Integrations team labels Nov 28, 2019
@mtojek

This comment has been minimized.

@mtojek
Copy link
Contributor Author

mtojek commented Nov 28, 2019

jenkins, test this please

@mtojek mtojek changed the title [Filebeat] Create ActiveMQ module WIP: [Filebeat] Create ActiveMQ module Nov 28, 2019
@mtojek mtojek force-pushed the 14744-create-filebeat-module branch 2 times, most recently from d4d1767 to e7f3a21 Compare November 28, 2019 14:56
@mtojek
Copy link
Contributor Author

mtojek commented Nov 28, 2019

Verified that issues reported by Travis CI are not related. Rebased against master.

Here is another PR failing similarly: https://travis-ci.org/elastic/beats/builds/618182978?utm_source=github_status&utm_medium=notification . Most likely there is regretion introduced.

@mtojek mtojek changed the title WIP: [Filebeat] Create ActiveMQ module [Filebeat] Create ActiveMQ module Nov 28, 2019
@mtojek mtojek force-pushed the 14744-create-filebeat-module branch 3 times, most recently from 7c0a62d to dd6096e Compare November 28, 2019 19:56
@codecov
Copy link

codecov bot commented Nov 28, 2019

Codecov Report

Merging #14840 into master will increase coverage by 0.02%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #14840      +/-   ##
==========================================
+ Coverage   58.71%   58.73%   +0.02%     
==========================================
  Files         545      545              
  Lines       34754    34754              
==========================================
+ Hits        20406    20413       +7     
+ Misses      12440    12432       -8     
- Partials     1908     1909       +1
Impacted Files Coverage Δ
metricbeat/module/couchbase/node/data.go 97.5% <0%> (-2.5%) ⬇️
filebeat/input/log/input.go 46.34% <0%> (ø) ⬆️
auditbeat/module/file_integrity/config.go 100% <0%> (+4.16%) ⬆️
auditbeat/module/file_integrity/action.go 80.7% <0%> (+12.28%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5fa01c...dd6096e. Read the comment docs.

@mtojek mtojek force-pushed the 14744-create-filebeat-module branch from dd6096e to a32eb07 Compare November 29, 2019 07:25
@mtojek
Copy link
Contributor Author

mtojek commented Nov 29, 2019

I reckon that errors reported for "upgrade from an older release" are irrelevant.

@mtojek
Copy link
Contributor Author

mtojek commented Nov 29, 2019

jenkins, test this please

Copy link
Member

@ChrsMark ChrsMark left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking the time to check the suggestion!

/lgtm

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking great! I have added some suggestions, nothing serious.

@mtojek mtojek force-pushed the 14744-create-filebeat-module branch from a913484 to 2863313 Compare November 29, 2019 20:07
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@mtojek
Copy link
Contributor Author

mtojek commented Dec 1, 2019

I renamed activemq.audit.action to message to properly present logs in default Kibana view. Otherwise, Kibana reports a missing field error.

Copy link
Member

@jsoriano jsoriano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@mtojek mtojek merged commit d4a6086 into elastic:master Dec 2, 2019
@mtojek mtojek added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 3, 2019
mtojek added a commit to mtojek/beats that referenced this pull request Dec 3, 2019
* Enable audit logs in ActiveMQ module

* Generate module stub

* Update configuration

* Add log files

* Add sample exception to ActiveMQ log

* Define ingest pipeline for ActiveMQ audit logs

* Define ingest pipeline for ActiveMQ logs

* Adjust ingest pipelines, bugfixing

* Do not check timestamp for audit logs

* Refactor fields

* Fix: mage fmt update

* Adjust fields after review

* Rename action to message to properly present logs in Kibana

* Refactor fields according to the review

(cherry picked from commit d4a6086)

Rename
mtojek added a commit that referenced this pull request Dec 3, 2019
* [Filebeat] Create ActiveMQ module (#14840)

* Enable audit logs in ActiveMQ module

* Generate module stub

* Update configuration

* Add log files

* Add sample exception to ActiveMQ log

* Define ingest pipeline for ActiveMQ audit logs

* Define ingest pipeline for ActiveMQ logs

* Adjust ingest pipelines, bugfixing

* Do not check timestamp for audit logs

* Refactor fields

* Fix: mage fmt update

* Adjust fields after review

* Rename action to message to properly present logs in Kibana

* Refactor fields according to the review

(cherry picked from commit d4a6086)

Rename

* Update CHANGELOG
@kaiyan-sheng
Copy link
Contributor

@mtojek @andresrc This PR also should be included in the test plan I believe.

@mtojek mtojek added the test-plan Add this PR to be manual test plan label Jan 14, 2020
@ChrsMark ChrsMark self-assigned this Jan 16, 2020
@ChrsMark
Copy link
Member

ChrsMark commented Jan 17, 2020

Tested module manually full stack (including dashboards) and works as expected.

@ChrsMark ChrsMark added test-plan-ok This PR passed manual testing and removed test-plan Add this PR to be manual test plan labels Jan 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
in progress Pull request is currently in progress. module review Team:Integrations Label for the Integrations team test-plan-ok This PR passed manual testing v7.6.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants