[Filebeat] Add RabbitMQ filebeat module #12032

jfsiii · 2019-05-02T16:43:25Z

Sorry if you got pulled in for review b/c of my git issues

I'm still getting used to a git rebase workflow

Summary

Accepts single and multiline messages from RabbitMQ log files
Uses ECS fields timestamp, log.level, and message
Followed convention for supporting converting to UTC via var.convert_timezone (defaults to false)
Added changelog.next entry in Added > Filebeat

Checklist from #11692

Tests

‣ MODULES_PATH=./module INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=rabbitmq nosetests tests/system/test_xpack_modules.py -v
test_fileset_file_0_rabbitmq (test_xpack_modules.XPackTest) ... ok

----------------------------------------------------------------------
Ran 1 test in 3.320s

OK

closes #11692

ruflin

This looks already very good. I left a few minor comments but I'm aware it's still WIP.

Could you also add a changelog to Changelog.next.asciidoc?

x-pack/filebeat/module/rabbitmq/_meta/config.yml

x-pack/filebeat/module/rabbitmq/log/_meta/fields.yml

x-pack/filebeat/module/rabbitmq/log/manifest.yml

x-pack/filebeat/module/rabbitmq/log/test/test.log

ruflin · 2019-05-06T12:13:01Z

I was checking locally on why this PR is failing on CI. I got the following diff:

     {
-        "@timestamp": "2019-04-03T11:13:15.076-04:00",
+        "@timestamp": "2019-04-03T11:13:15.076+02:00",
         "ecs.version": "1.0.0",
         "event.dataset": "rabbitmq.log",
         "event.module": "rabbitmq",
-        "event.timezone": "-04:00",
+        "event.timezone": "+02:00",
         "fileset.name": "log",
         "input.type": "log",
         "log.level": "info",
@@ -14,11 +14,11 @@
         "service.type": "rabbitmq"
     },
...

Obviously the problem is that you, mean and CI are not in the same time zone. I think we didn't hit this issue in the past we made the time zone conversion optional. @exekias Any thoughts around this?

jfsiii · 2019-05-06T20:10:11Z

RabbitMQ logs are built on Lager which "by default, formats timestamps as local time for whatever computer generated the log message."

e.g.
https://github.com/elastic/beats/blob/40f7ce7893b1932b57dc4f1c84d2ebcf0ed0902a/x-pack/filebeat/module/rabbitmq/log/test/test.log#L68-L78

Notice the lack of timezone name or offset.

I wanted these timestamps ingested/stored as UTC but wasn't sure how/where best to do the transform. I saw a few config/${fileset}.yml files conditionally add timezone

processors:
{{ if .convert_timezone }}
  - add_locale: ~
{{ end }}

but I went with always on to keep moving and because of my questions about how/where to transform.

Another option is for this to follow the convert_timezone pattern (and default to enabled). Let me know if there are preferred/common approach to this.

ruflin · 2019-05-07T08:39:33Z

This goes into the direction of #11858 that it should be on by default.

We need to find a way that the tests always generate the same output. I think there are multiple options here. One would be to "fake" the timezone in which the Beat is running to make sure it's for example always UTC. We could do this by for example allowing add_locale a param to set the local timezone / overwrite the auto detection and use this in the tests.

As the above needs further changes what I can see two steps forward with this PR:

Follow the example of all the existing modules and have it off by default (not optimal)
Have the condition to enabled it and have it enabled by default. The tricky part with this is then that we need to disable it somehow for the tests, which probably means a hack in the test_modules file (we already have a few of these ...).

jfsiii · 2019-05-08T14:42:47Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.json

+      "date" : {
+        "field" : "timestamp",
+        "formats" : ["yy-MM-dd HH:mm:ss.SSS"],
+        {< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}


I copied this from other modules but have no idea how {< if .convert_timezone >} works. Are there any docs about this syntax?

I assume it's a know issue/tradeoff but this makes the file invalid as JSON.

Yes, it's a know tradeoff as it's the way that makes it possible for us to have pipeline templates. Recently a few modules started to use yaml instead of json but I assume the above template language will also make it invalid yaml. @adriansr might know?

Pipelines and configuration files for a module are preprocessed with Golang template package (Using {< >} instead of {{ }}as delimiters).

I didn't had a problem with the files being valid JSON/YAML because my editor doesn't complain too much. But if this is a problem, you can switch the pipeline to YAML and hide the actions in a comment:

- date: field: "_temp_.generated_time" ignore_failure: true #{< if .convert_timezone >} timezone: "{{ event.timezone }}" #{< end >} formats: - "yyyy/MM/dd HH:mm:ss"

Nice trick with the commends in yaml. I like it.

jfsiii · 2019-05-08T14:50:28Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.json

+      "grok" : {
+        "field" : "message",
+        "pattern_definitions" : {
+          "GREEDYMULTILINE" : "(.|\n)*",


This seems useful to other beats. Think it's worth a PR to https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns ?

So far we never contributed patterns back to the grok patterns. I kind of like the idea but don't know how grok pattern etc. are versioned and then make it into the products. @jsvd might know more here.

Additional note: The grok patterns you linked above are the ones used by Logstash, I assume there is a different link for Elasticsearch.

Re: Additional note
Here's the patterns used by the ingest processor:
https://github.com/elastic/elasticsearch/blob/master/libs/grok/src/main/resources/patterns/grok-patterns

ruflin · 2019-05-09T07:37:53Z

@jfsiii Could you rebase on master. This should resolve your CI issues (mostly).

exekias · 2019-05-09T08:59:36Z

I was checking locally on why this PR is failing on CI. I got the following diff:
     {
-        "@timestamp": "2019-04-03T11:13:15.076-04:00",
+        "@timestamp": "2019-04-03T11:13:15.076+02:00",
         "ecs.version": "1.0.0",
         "event.dataset": "rabbitmq.log",
         "event.module": "rabbitmq",
-        "event.timezone": "-04:00",
+        "event.timezone": "+02:00",
         "fileset.name": "log",
         "input.type": "log",
         "log.level": "info",
@@ -14,11 +14,11 @@
         "service.type": "rabbitmq"
     },
...
Obviously the problem is that you, mean and CI are not in the same time zone. I think we didn't hit this issue in the past we made the time zone conversion optional. @exekias Any thoughts around this?

we discussed this one offline, we can try to set the timezone in the tests, so everyone uses the same when running them, @ruflin is taking a stab at this

Currently all our modules have convert_timezone disable by default. The reason in 6.x for this was probably that 6.0 did not support convert_timezone and we did not want to introduce a breaking changes. New modules should have convert_timezone enabled by default. If a module has convert_timezone enabled by default the tests will fail as it takes the timezone of the local computer. To circumvent this, this PR sets the timezone of the tests to UTC so the same time zone is always used. No generated files were changed in this PR as all modules have convert_timezone off by default. But it will affect elastic#12079 and elastic#12032

ruflin · 2019-05-09T09:14:26Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.json

+      "date" : {
+        "field" : "timestamp",
+        "formats" : ["yy-MM-dd HH:mm:ss.SSS"],
+        {< if .convert_timezone >}"timezone": "{{ event.timezone }}",{< end >}


@exekias Should we still allow to disable the time zone conversion? I don't really see an option to make it optional and with #12120 it's not going to be a problem anymore for the tests.

I think it would make sense to hardcode this for new modules, we can always add it later if we see a demand for it

libbeat/processors/extract_array/extract_array.go

libbeat/processors/extract_array/extract_array_test.go

libbeat/processors/extract_array/extract_array.go

Currently all our modules have convert_timezone disable by default. The reason in 6.x for this was probably that 6.0 did not support convert_timezone and we did not want to introduce a breaking changes. New modules should have convert_timezone enabled by default. If a module has convert_timezone enabled by default the tests will fail as it takes the timezone of the local computer. To circumvent this, this PR sets the timezone of the tests to UTC so the same time zone is always used. No generated files were changed in this PR as all modules have convert_timezone off by default. But it will affect #12079 and #12032

ruflin · 2019-05-10T06:17:18Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.yml

@@ -0,0 +1,27 @@
+---
+description: Pipeline for parsing RabbitMQ logs


Nice, you converted it to YAML 👍

Yeah 😄 I started because of #12032 (comment) but I also realized it was the only config file in JSON. Now the only JSON file is the -expected.json

ruflin · 2019-05-10T06:17:47Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.yml

+    formats:
+    - yy-MM-dd HH:mm:ss.SSS
+    ignore_failure: true
+    #{< if .convert_timezone >}


As this module is only file based at the moment, should we skip the convert timezone completely and have it always enabled until requested otherwise by users?

I figured I'd keep it to be consistent with the others and allow the user to disable it if they choose.

I think enabling it by default on this module makes sense b/c the default log is localtime so I'm going to do that and make sure everything still works as expected.

ruflin

Went through it again and only some nits.

I would say lets get this in rather soonish and do several follow up PR's. WDYT @jfsiii ?

filebeat/docs/modules/rabbitmq.asciidoc

ruflin · 2019-05-10T13:38:03Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.yml

+      %{GREEDYMULTILINE:message}"
+    ignore_missing: true
+- date:
+    field: timestamp


I somehow miss the part where it's converted to @timestamp but based on the output below it works. I'm confused.

field is the input field. target_field is the output field and it defaults to @timestamp

I omitted it because I want the default behavior, but I can make it explicit if that's preferable.

Good to know, somehow missed that default. I'm on the fence on this one. I really like that @timestamp is the default because of ECS. At the same time having it in here would make it more obvious what is happening for an other engineering touching this. I'm good with both.

x-pack/filebeat/module/rabbitmq/module.yml

ruflin

LGTM

Ready to be merged 🎉 Please squash it into 1 commit when merging.

ruflin · 2019-05-15T10:33:43Z

filebeat/docs/include/var-convert-timezone.asciidoc

@@ -5,4 +5,6 @@ parsing time to convert the timestamp to UTC. The local timezone is also added
 in each event in a dedicated field (`beat.timezone`). The conversion is only
 possible in Elasticsearch >= 6.1. If the Elasticsearch version is less than 6.1,
 the `beat.timezone` field is added, but the conversion to UTC is not made.  The
-default is `false`.
+default is
+ifdef::default_convert_timezone[`true`]


Nice addition.

ruflin · 2019-05-15T10:35:26Z

x-pack/filebeat/module/rabbitmq/log/ingest/pipeline.yml

+      %{GREEDYMULTILINE:message}"
+    ignore_missing: true
+- date:
+    field: timestamp


Good to know, somehow missed that default. I'm on the fence on this one. I really like that @timestamp is the default because of ECS. At the same time having it in here would make it more obvious what is happening for an other engineering touching this. I'm good with both.

"works on my machine" but lots of TODOs remain Took a shot at ECS. Tested with ``` ~/go/src/github.com/elastic/beats/x-pack/filebeat [master @ 092a3f5] ✓ ‣ MODULES_PATH=./module GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=rabbitmq nosetests tests/system/test_xpack_modules.py -v test_fileset_file_0_rabbitmq (test_xpack_modules.XPackTest) ... ok ---------------------------------------------------------------------- Ran 1 test in 3.320s OK ```

Update fields and docs.

Co-Authored-By: jfsiii <github.com@jfsiii.org>

* add missing test.log-expected.json * remove aliases * re-run `mage update` and `mage fmt update`

Use format detailed in https://www.elastic.co/guide/en/beats/filebeat/master/using-environ-vars.html

Unrelated whitespace change

This is also consistent with other filebeats.

…ic#12120) Currently all our modules have convert_timezone disable by default. The reason in 6.x for this was probably that 6.0 did not support convert_timezone and we did not want to introduce a breaking changes. New modules should have convert_timezone enabled by default. If a module has convert_timezone enabled by default the tests will fail as it takes the timezone of the local computer. To circumvent this, this PR sets the timezone of the tests to UTC so the same time zone is always used. No generated files were changed in this PR as all modules have convert_timezone off by default. But it will affect elastic#12079 and elastic#12032

* Parses single and multiline messages from [RabbitMQ's unified log file](https://www.rabbitmq.com/logging.html#log-file-location) * `var.paths` uses the `RABBITMQ_LOGS` env variable if present * `var.convert_timezone` (enabled by default since the [timestamps are localtime by default](https://github.com/erlang-lager/lager#universal-time)) * No dashboard * Tested with ``` ~/go/src/github.com/elastic/beats/x-pack/filebeat [master @ 092a3f5] ✓ ‣ MODULES_PATH=./module GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=rabbitmq nosetests tests/system/test_xpack_modules.py -v test_fileset_file_0_rabbitmq (test_xpack_modules.XPackTest) ... ok ---------------------------------------------------------------------- Ran 1 test in 3.320s OK ```

* Use time.Duration directly in GetStartTimeEndTime function (#12033) * Remove convertPeriodToDuration and use duration directly in GetStartTimeEndTime * Pass period in time.Duration type * Fix memory leak in Filebeat pipeline acker (#12063) * Fix memory leak in Filebeat pipeline acker Before this change acker goroutine was kept forever as processed events count was not correctly updated. Filebeat sends an empty event to update file states, this event is not published, but treated as dropped, without updating counters. This change makes sures that `a.events` count gets updated for dropped events also, so the acker gets closed after all ACKs happen. * Add convert processor (#11686) The `convert` processor converts a field in the event to a different type, such as converting a string to an integer. For a full description of the processor's capabilities see the included documentation. Closes #8124 * Update docs.asciidoc (#11852) (#12045) * Update docs.asciidoc Added in a short note on a few of the other configurable variables. * Make update * Remove systemd v233 requirement because it's no longer true (#12076) * Don't generate autodiscover config when no port matches host hints (#12086) On metricbeat, when the host autodiscover hint is used, and it includes the port, one of the exposed ports has to match with the one in the hint. If not, no configuration should be generated. If it is generated, it will have empty hosts, what would lead to unexpected errors as the seen in #8264. * Fix transptest testing (#12091) Rewrite test for more clarity ensuring that the server and listeners are stopped before the test function returns (cause for the case condition). Each test has it's very own socks5 server now. Tests not requiring a proxy don't spin up one. * Refactor and add tests for template and ilm handling. (#12065) Refactor and add more tests related to setting up template and ilm policy. Ensure template and ilm-policy is exported despite enabled=false setting. * [Auditbeat] Login: Fix re-read of utmp files (#12028) The `login` dataset is not using the previous file offset when reading new entries in a utmp file. As a result, whenever a new login event occurs, all records are re-read. Also expands the documentation, moves test files to testdata/, and adds a test case that adds a utmp record to the test file and re-reads it to make sure this bug does not happen again. * [Metricbeat][postgresql] Update lib/pq to fix #11393 (#12094) This updates github.com/lib/pq to the latest version to fix a socket leak when SSL is not enabled on the PostgreSQL server. Fixes #11393 * Make breaking changes separate files (#12002) * Change image references to use block syntax not inline (#11911) * Change image references to use block syntax not inline * Fix format for inline links * Revert "Make breaking changes separate files (#12002)" (#12116) This reverts commit c17586a. * fix queue.spool.write.flush.events config type (#12080) * Revert printing template and policy name on export. (#12067) Ensures behavior on running the `export` cmd does not change compared to last released version. * [Filebeat] Add -expected files by default (#12041) So far expected files in Filebeat tests were only generated and compared when a file exists. This changes to create a generated for all example logs. This will add a few more files to the repository but I think there the benefits outweight the costs as it means the modules are tested in more detail. Also minor changes will be detected easier. * Update vendored gosigar to 0.10.2 (#12101) #11924 duplicated some code from go-sysinfo that is affected by a memory leak (fixed in #12100) In this case, only master is affected as the PR that introduced the leaky gosigar wasn't backported. * Fix various memory leaks under Windows (#12100) A function in go-sysinfo used under Windows to split a command-line into arguments was leaking memory. This code was used in various places: add_process_metadata processor. (enabled by default in all Beats) Packetbeat's process monitor. (disabled by default). Auditbeat's system/process metricset. This PR updates vendored go-sysinfo to the most recent version, which fixes the leak. * New processor extract_array (#11761) This adds a new processor, extract_array, that allows accessing values inside arrays and copying them to target fields. * [Heartbeat] Remove not needed flags from setup command (#11856) The setup command until now contained all the possible options from the other Beats. As Heartbeat does not ship anymore with dashboards, the --dashboards command is not needed anymore and is only confusing. I also removed all the other commands except `--ilm-policy` and `--template`. I'm not aware that `--pipelines` or `--machine-learning` would be used. Here the comparison between `./heartbeat setup -h` from before and after. Before: ``` This command does initial setup of the environment: * Index mapping template in Elasticsearch to ensure fields are mapped. * Kibana dashboards (where available). * ML jobs (where available). * Ingest pipelines (where available). * ILM policy (for Elasticsearch 6.5 and newer). Usage: heartbeat setup [flags] Flags: --dashboards Setup dashboards -h, --help help for setup --ilm-policy Setup ILM policy --machine-learning Setup machine learning job configurations --pipelines Setup Ingest pipelines --template Setup index template ``` After: ``` This command does initial setup of the environment: * Index mapping template in Elasticsearch to ensure fields are mapped. * ILM Policy Usage: heartbeat setup [flags] Flags: -h, --help help for setup --ilm-policy Setup ILM policy --template Setup index template ``` In this PR I did not include a check for the config option `setup.dashboards` to make sure they are not there like apm-server does (https://github.com/elastic/apm-server/blob/2baefab778fdfe70c47bc2fb488677b2e43e4635/beater/beater.go#L60) as I don't think it's necessary. * Skip Windows testing if magefile.go does not exist (#12099) Changes the jenkins_ci.ps1 script to skip testing when magefile.go does not exist. This will allow us to add projects like x-pack/winlogbeat to the test matrix because not all branches have an x-pack/winlogbeat/magefile.go file. * Fix goroutine leak on initialization failures of log input (#12125) Outlets are created during log input initialization, and if it fails they were never freed. Handle this case. * Document and improve permission checks when running socket metricset from Docker (#12039) Update instructions for system/socket metricset on Docker. And base permission checks on capabilities rather than on the effective uid. Running a process as root doesn't mean that it has all privileges, specially when run as container. * [metricbeat] added CPU usage check to docker memory stats (#12062) * Change type from scaled_float to long and add format (#11982) * Change type from scaled_float to long and add format * [libbeat] Add unit tests for libbeat's client proxy settings (#12044) These tests set up server listeners and create libbeat clients with varying proxy settings, and verify that the clients ping the correct target URL. This is a preparation for #11713, since most of the logic (and work) is in testing the proxy settings; the much simpler PR adding the proxy-disable flag will be a followup to this one, to keep the functional changes isolated in case of rollbacks etc. * [Metricbeat](Etcd-Leader)Followers wont report leader metrics (#12004) * manage leader metricset so that followers don't report errors nor events * add debug message when skipping leader events from non leader members * Add package libbeat/common/cleanup (#12134) * Add package libbeat/common/cleanup The cleanup package adds helpers for deferred optional cleanup on errors. For example: ``` ok := False defer cleanup.IfNot(&ok, func() { ... }) // continue initialization ok = True return // some value ``` * Add changelog entry * [docs] add make fmt to contributing guide (#12118) * Move one changelog entry from breaking change to bug fix (#12146) * Sysmon and Security "modules" for Winlogbeat (#11651) Add pipelines for the Security and Sysmon event logs The Security processor handles just three events to start with - 4624, 4625, and 4648. These are event.category=authentication events. The Sysmon processor handles all event IDs found in the sysmon manifest (sysmon -s). It moves the event_data fields that are in ECS and does some type conversions. * [Auditbeat] Process: Add hash of executable (#11722) Adds the hash(es) of the process executable to `process.hash.*`. The default is to add SHA-1 only as `process.hash.sha1`. * [Docs] Comment out section that contains bad link (#12152) * [Filebeat] Introduce UTC as default timezone for modules tests (#12120) Currently all our modules have convert_timezone disable by default. The reason in 6.x for this was probably that 6.0 did not support convert_timezone and we did not want to introduce a breaking changes. New modules should have convert_timezone enabled by default. If a module has convert_timezone enabled by default the tests will fail as it takes the timezone of the local computer. To circumvent this, this PR sets the timezone of the tests to UTC so the same time zone is always used. No generated files were changed in this PR as all modules have convert_timezone off by default. But it will affect #12079 and #12032 * Add number of goroutines to reported metrics (#12135) * Add minimal ES template functionality. (#12103) When loading a template without fields, create a minimal template only applying given configuration, without any default values for mappings and settings. This allows to create additional templates only defining specific values. * Refactor logging in pgsql module (#12151) Guard debug logging statements with "isDebug" checks. And switch the module over to using named loggers. Fixes #12150 * Ignore doc type in ES search API for ES 8 (#12171) * [Docs] Make breaking changes separate files for each version (#12173) * [Filebeat] module for palo_alto (pan-os) logs (#11999) This is a module for Palo Alto Networks PAN-OS logs received via Syslog. It has been tested with logs for PAN-OS version 7.1 to 9.0. However, it is expected to work with earlier versions as the log format is compatible. * [cmd setup] Add and deprecate setup cmds for index handling (#12132) Deprecate `setup --template` and `setup --ilm` in favour of newly introduced `setup --index-management` command. Fix bug in template and write alias creation order to ensure creating properly managed indices. implements #12095 Co-authored-by: steffen.siering@elastic.co * Zdd zfs beat (#12136) Querying ZFS Storage and Pool Status * [metricbeat] Expand metricbeat dev guide for testing (#12105) * update dev guide to add examples and expand testing * Bugfix set template.order to 1 by default. (#12160) To ensure default order is not changed to 0, set it to 1 in default config. * [Auditbeat] Fix issues with multiple calls to rpmReadConfigFiles (#12168) This patch fixes two issues in Auditbeat's system/package on RPM distros: - Multiple calls to rpmReadConfigFiles lead to a crash (segmentation fault). It is necessary to call rpmFreeRpmrc after each call to rpmReadConfigFiles. See [1] for a similar issue. - In addition, it is also necessary to call rpmFreeMacros (when available) to avoid leaking memory after each rpmReadConfigFiles call. 1: https://lists.fedorahosted.org/pipermail/anaconda-patches/2015-February/015826.html Fixes #12147 * [Filebeat] Palo_alto module improvements (#12182) This PR adds some missing features to the recently merged palo_alto module: Dashboards (One for traffic logs, one for threats). Sets network.type to either ipv4 or ipv6. Renames palo_alto.pan_os.threat_file_or_url to palo_alto.pan_os.threat.resource. Splits palo_alto.pan_os.threat_id into palo_alto.pan_os.threat.id and palo_alto.pan_os.threat.name. * Add mesosbeat to the community beats (#12185) * Missing module.yml.disabled file for palo_alto (#12191) * [Filebeat] Add RabbitMQ module (#12032) * Parses single and multiline messages from [RabbitMQ's unified log file](https://www.rabbitmq.com/logging.html#log-file-location) * `var.paths` uses the `RABBITMQ_LOGS` env variable if present * `var.convert_timezone` (enabled by default since the [timestamps are localtime by default](https://github.com/erlang-lager/lager#universal-time)) * No dashboard * Tested with ``` ~/go/src/github.com/elastic/beats/x-pack/filebeat [master @ 092a3f5] ✓ ‣ MODULES_PATH=./module GENERATE=1 INTEGRATION_TESTS=1 TESTING_FILEBEAT_MODULES=rabbitmq nosetests tests/system/test_xpack_modules.py -v test_fileset_file_0_rabbitmq (test_xpack_modules.XPackTest) ... ok ---------------------------------------------------------------------- Ran 1 test in 3.320s OK ``` * [metricbeat] Add linux sockstat data to socket_summary metricset (#12050) * add sockstat data to socket_summary * Fix goroutine leak on non-explicit finalization of log inputs (#12164) If log inputs were finished because their context, or one of their ouleters have been finished, then it wasn't stopping its harvesters, leaking resources. * [Metricbeat] CoreDNS module: Add Kibana Dashboard (#11619) * [Metricbeat] CoreDNS module: Add Kibana Dashboard Fixes #10432. * [Libbeat][Metricbeat]Add IgnoreAllErrors to schema.Conv object (#12089) * schema conv option for ignore all errors Co-Authored-By: Jaime Soriano Pastor <jaime.soriano@elastic.co> * Add `container` input, deprecate `docker` in favor of it (#12162) Add `container` input, deprecate `docker` in favor of it This change adds a new container input for better support of CRI based scenarios. `docker` input was acting as a catch all input for all container related cases, but its config options were very opinionated towards Docker, with some issues: * `containers.ids` setting was good to abstract logs path, but we have seen many cases were logs are not under default location, or follow a different path pattern (ie CRI logs). * `containers.*` settings have shown counter intuitive for many users, in many cases we have seen people writing `container.*` instead, ending up in a config error. * Some existing settings (`combine_partials`, `cri.parse_flags`) were introduced as a way to offer a backwards compatible upgrades, but it doesn't really make sense to disable them, as they handle actual format behaviors. This new `container` input offers the same wrapper to read log files from containers with the following changes: * It exposes `paths` as the `log` input, instead of `containers.ids` and `containers.path`. * `parse_flags` and `combine_partials` are hardcoded, as there is no good reason to disable them. * `stream` selector is still available, under root settings. * It allows to select the log format (also atodetect it), giving room for future format changes. `format` can be `auto` (default), `docker` and `CRI`. Example configurations: Get Docker logs: ``` filebeat.inputs: - type: container paths: - /var/lib/docker/containers/*/*.log ``` Get Kubernetes logs: ``` filebeat.inputs: - type: container paths: - /var/log/pods/*/*/*.log # this could also be used: #- /var/log/containers/*.log ``` Previous `docker` input is deprecated in favor of this, to be removed in 8.0 * [Auditbeat] Fix formatting of config files on macOS and Windows (#12148) Fixes formatting of auditbeat.yml and auditbeat.reference.yml across platforms. * [libbeat] Escape BOM on JsonReader before trying to decode line (#11661) * fix json bom + testing * Set beat ID in registries after loading meta file (#12180) * Reset beat ID in registries in case loaded from meta file * Set, not reset * Adding CHANGELOG entry * [Filebeat] Move dashboards from 8 to 7 directory (#12217) The CoreDNS and Envoyproxy dashboard were in the 8 instead of 7 directory. This PR fixes this. * adjust doc

jfsiii requested a review from ruflin May 2, 2019 16:43

jfsiii requested review from a team as code owners May 2, 2019 16:43

jfsiii added Team:Integrations Label for the Integrations team Filebeat Filebeat module review labels May 2, 2019

ruflin changed the title ~~[RFC/WIP] Add RabbitMQ filebeat under x-pack~~ [RFC/WIP] Add RabbitMQ filebeat module May 3, 2019

ruflin reviewed May 3, 2019

View reviewed changes

ruflin added the in progress Pull request is currently in progress. label May 3, 2019

andresrc added the [zube]: In Progress label May 6, 2019

ruflin mentioned this pull request May 8, 2019

[Filebeat] MSSQL module #12079

Merged

jfsiii commented May 8, 2019

View reviewed changes

jfsiii changed the title ~~[RFC/WIP] Add RabbitMQ filebeat module~~ [Filebeat] Add RabbitMQ filebeat module May 8, 2019

ruflin mentioned this pull request May 9, 2019

[Filebeat] Introduce UTC as default timezone for modules tests #12120

Merged

ruflin reviewed May 9, 2019

View reviewed changes

jfsiii requested review from a team as code owners May 9, 2019 14:59

houndci-bot reviewed May 9, 2019

View reviewed changes

libbeat/processors/extract_array/extract_array.go Outdated Show resolved Hide resolved

houndci-bot reviewed May 9, 2019

View reviewed changes

libbeat/processors/extract_array/extract_array_test.go Outdated Show resolved Hide resolved

libbeat/processors/extract_array/extract_array.go Outdated Show resolved Hide resolved

jfsiii force-pushed the filebeat-rabbitmq branch 2 times, most recently from d98406c to 25471fe Compare May 9, 2019 15:37

ruflin reviewed May 10, 2019

View reviewed changes

jfsiii force-pushed the filebeat-rabbitmq branch from 7b55bf8 to bd19186 Compare May 10, 2019 16:38

ruflin approved these changes May 15, 2019

View reviewed changes

John Schulz and others added 15 commits May 16, 2019 12:27

Add alias from ECS fields to rabbitmq.log.*

896545b

Update fields and docs.

Run mage fmt update as requested by CI

4499b09

Update x-pack/filebeat/module/rabbitmq/_meta/config.yml

243498d

Co-Authored-By: jfsiii <github.com@jfsiii.org>

Update x-pack/filebeat/module/rabbitmq/_meta/config.yml

69a6353

Co-Authored-By: jfsiii <github.com@jfsiii.org>

Address feedback

89ed898

* add missing test.log-expected.json * remove aliases * re-run `mage update` and `mage fmt update`

Add changelog.next entry in Added > Filebeat

0a9bebb

Add config examples. Format links correctly.

30e4571

Follow convention for adding timezone via var.convert_timezone

63fcd83

If present, use RABBITMQ_LOGS env variable for var.paths setting

7dc222e

Use format detailed in https://www.elastic.co/guide/en/beats/filebeat/master/using-environ-vars.html

No dashboard for now.

419c7dc

Use YML for pipeline config

c762cc9

Unrelated whitespace change

Whitespace changes missed in 2fbbe2eb

f5d901f

PR feedback. Remove module.yml & note about MacOS

734359a

TZ default on for this module. Update docs & tests

cb8e632

jfsiii force-pushed the filebeat-rabbitmq branch from db1d043 to cb8e632 Compare May 16, 2019 16:33

Be explicit when creating @timestamp field.

013cf0b

This is also consistent with other filebeats.

jfsiii merged commit b5c92a7 into elastic:master May 16, 2019

zube bot added [zube]: Done and removed [zube]: In Progress labels May 16, 2019

andresrc removed the [zube]: Done label May 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Add RabbitMQ filebeat module #12032

[Filebeat] Add RabbitMQ filebeat module #12032

jfsiii commented May 2, 2019 •

edited by zube bot

Loading

ruflin left a comment

ruflin commented May 6, 2019

jfsiii commented May 6, 2019

ruflin commented May 7, 2019

jfsiii May 8, 2019

ruflin May 9, 2019

adriansr May 9, 2019 •

edited

Loading

ruflin May 9, 2019

jfsiii May 8, 2019

ruflin May 9, 2019

adriansr May 9, 2019

ruflin commented May 9, 2019

exekias commented May 9, 2019

ruflin May 9, 2019

exekias May 9, 2019

ruflin May 10, 2019

jfsiii May 10, 2019

ruflin May 10, 2019

jfsiii May 10, 2019

ruflin May 10, 2019

ruflin left a comment

ruflin May 10, 2019

jfsiii May 13, 2019

ruflin May 15, 2019

ruflin left a comment

ruflin May 15, 2019

ruflin May 15, 2019

		@@ -0,0 +1,27 @@
		---
		description: Pipeline for parsing RabbitMQ logs

[Filebeat] Add RabbitMQ filebeat module #12032

[Filebeat] Add RabbitMQ filebeat module #12032

Conversation

jfsiii commented May 2, 2019 • edited by zube bot Loading

Sorry if you got pulled in for review b/c of my git issues

Summary

Checklist from #11692

Tests

ruflin left a comment

Choose a reason for hiding this comment

ruflin commented May 6, 2019

jfsiii commented May 6, 2019

ruflin commented May 7, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr May 9, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin commented May 9, 2019

exekias commented May 9, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ruflin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jfsiii commented May 2, 2019 •

edited by zube bot

Loading

adriansr May 9, 2019 •

edited

Loading