[Filebeat] googlecloud module indexing error #18465
Labels
Filebeat
Filebeat
Comments
botelastic
bot
added
the
needs_team
|
Pinging @elastic/siem (Team:SIEM) |
botelastic
bot
removed
the
needs_team
|
This problem was caused by some new fields introduced by the response. I am not exactly sure why the field "status" is considered an object since googlecloud.audit module does not define this field under the group "response". I will add the new fields and handle them explicitly. |
|
@alakahakai there may be some concerns with publicly providing raw audit logs due to data sanitization issues. Is there an easy way to retrieve the raw audit logs? These are coming from Pub/Sub, so I am not sure the best way to retrieve them prior to any processing. |
|
I see. Not a problem then. I will try to see whether I can get some additional raw data from our test setup. Thanks. |
alakahakai
pushed a commit
that referenced
this issue
May 14, 2020
…og (#18472) * Fix issue 18465 * Update test data for googlecloud.audit dataset * Update CHANGELOG.next.asciidoc
v1v
added a commit
to v1v/beats
that referenced
this issue
May 15, 2020
…w-oss * upstream/master: (27 commits) Disable host fields for "cloud", panw, cef modules (elastic#18223) [docs] Rename monitoring collection from legacy internal collection to legacy collection (elastic#18504) Introduce auto detection of format (elastic#18095) Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472) Fix libbeat import path in seccomp policy template (elastic#18418) Address Okta input issue elastic#18530 (elastic#18534) [Ingest Manager] Avoid Chown on windows (elastic#18512) Fix Cisco ASA/FTD msgs that use a host name as NAT address (elastic#18376) [CI] Optimise stash/unstash performance (elastic#18473) Libbeat: Remove global loggers from libbeat/metric and libbeat/cloudid (elastic#18500) Fix PANW bad mapping of client/source and server/dest packets and bytes (elastic#18525) Add a file lock to the data directory on startup to prevent multiple agents. (elastic#18483) Followup to 12606 (elastic#18316) changed input from syslog to tcp/udp due to unsupported RFC (elastic#18447) Improve ECS field mappings in Sysmon module. (elastic#18381) [Elastic Agent] Cleaner output of inspect command (elastic#18405) [Elastic Agent] Pick up version from libbeat (elastic#18350) Update communitybeats.asciidoc (elastic#18470) [Metricbeat] Change visualization interval from 15m to >=15m (elastic#18466) docs: Fix typo in kerberos docs (elastic#18503) ...
|
It appears that this issue is still occurring in 7.8.0: |
|
Sorry @SpencerLN, this fix was never backported to 7.8. @alakahakai can you please backport this to 7.8 so that it is included in 7.8.1. |
|
@andrewkroh it looks like this is still occurring in 7.8.1: |
|
With a quick look, it appears that the processor that's moving the value into the keyword diff --git a/x-pack/filebeat/module/googlecloud/audit/config/pipeline.js b/x-pack/filebeat/module/googlecloud/audit/config/pipeline.js
index c467cc77c..8ab8750f1 100644
--- a/x-pack/filebeat/module/googlecloud/audit/config/pipeline.js
+++ b/x-pack/filebeat/module/googlecloud/audit/config/pipeline.js
@@ -85,7 +85,7 @@ function Audit(keep_original_message) {
{from: "json.requestMetadata", to: "googlecloud.audit.request_metadata"},
{from: "json.response.@type", to: "googlecloud.audit.response.proto_name"},
- {from: "json.response.status", to: "googlecloud.audit.response.status"},
+ {from: "json.response.status", to: "googlecloud.audit.response.status", type: "string"},
{from: "json.response.details.group", to: "googlecloud.audit.response.details.group"},
{from: "json.response.details.kind", to: "googlecloud.audit.response.details.kind"},
{from: "json.response.details.name", to: "googlecloud.audit.response.details.name"},
@@ -103,6 +103,7 @@ function Audit(keep_original_message) {
],
mode: "rename",
ignore_missing: true,
+ fail_on_error: false,
});I'll add some test cases and look for other similar issues. |
6 tasks
|
I have opened a PR to prevent the object from being copied into the keyword field. See #20465. |
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this issue
Aug 6, 2020
Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes elastic#18465
andrewkroh
added a commit
that referenced
this issue
Aug 6, 2020
Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes #18465
6 tasks
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this issue
Aug 6, 2020
) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes elastic#18465 (cherry picked from commit 924c239)
andrewkroh
added a commit
to andrewkroh/beats
that referenced
this issue
Aug 6, 2020
) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes elastic#18465 (cherry picked from commit 924c239)
6 tasks
andrewkroh
added a commit
that referenced
this issue
Aug 6, 2020
…20479) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes #18465 (cherry picked from commit 924c239)
andrewkroh
added a commit
that referenced
this issue
Aug 11, 2020
…20480) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes #18465 (cherry picked from commit 924c239)
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this issue
Oct 14, 2020
) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes elastic#18465
leweafan
pushed a commit
to leweafan/beats
that referenced
this issue
Apr 28, 2023
…sue elastic#18465 for googlecloud audit log (elastic#18554) * Add additional fields to address issue elastic#18465 for googlecloud audit log (elastic#18472)
leweafan
pushed a commit
to leweafan/beats
that referenced
this issue
Apr 28, 2023
) (elastic#20479) Specify type to the convert processor so that fields are renamed only if the value is (or can be coerced to) the desired type. This prevents the module from writing incompatible data types into fields and prevents mapping exceptions. The data in the google cloud audit logs can vary based on the source service so any time the module makes an assumption about a field that's not explicitly documented the module should be very defensive. The request/response object values would be good candidates for the flattened data type if we want to make a future change. Fixes elastic#18465 (cherry picked from commit 1b0cc03)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
While running the Filebeat googlecloud module we noticed an error messaging regarding events failing to be indexed to Elasticsearch in the logs:
Filebeat templates were installed prior to indexing data and have not been modified.
Configuration:
For confirmed bugs, please report:
The text was updated successfully, but these errors were encountered: