[Filebeat] [MongoDB] Support MongoDB 4.4 json logs #24774

tetianakravchenko · 2021-03-25T21:03:40Z

What does this PR do?

MongoDB 4.4 changed logs to a structured JSON format, this PR adds support for mongodb logs json format, and support the old format too.

Why is it important?

to support new logs format in filebeat MongoDB module.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

[ ]

How to test this PR locally

Related issues

Closes Support MongoDB 4.4 in filebeat's MongoDB module #20501

Use cases

Screenshots

Logs

elasticmachine · 2021-03-25T21:08:43Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24774 updated
Start Time: 2021-06-15T11:40:44.621+0000
Duration: 104 min 5 sec
Commit: be5576a

Test stats 🧪

Test	Results
Failed	0
Passed	14053
Skipped	2306
Total	16359

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14053
Skipped	2306
Total	16359

jsoriano · 2021-03-29T19:12:13Z

/test

elasticmachine · 2021-03-29T19:12:25Z

Pinging @elastic/integrations (Team:Integrations)

jsoriano · 2021-03-30T09:40:36Z

@tetianakravchenko thanks for this contribution!

Failure in tests seems related:

Pipeline processor configured for non-existent pipeline [filebeat-8.0.0-mongodb-log-pipeline-plaintext]

I think the problem is that new pipelines need to be added to the manifest in filebeat/module/mongodb/log/manifest.yml, could you please add the pipelines there?

tetianakravchenko · 2021-03-30T09:54:32Z

@jsoriano thank you for the feedback! I've added new pipelines to the manifest 9f5d34c

jsoriano · 2021-03-30T10:07:44Z

/test

filebeat/module/mongodb/_meta/docs.asciidoc

jsoriano · 2021-03-30T11:53:26Z

/test

cihantunali · 2021-05-28T06:40:18Z

Hi there,
I dont know if it helps or not but I created a GROK pattern to parse it. I tried to implement it via manuel way to filebeat (created new pipeline and forced filebeat to use it) but no luck :(

{"t":{"$date":"%{TIMESTAMP_ISO8601:timestamp}"},"s":"%{WORD:severity}",%{SPACE}"c":"%{WORD:component}",%{SPACE}"id":%{INT:id},%{SPACE}"ctx":"%{DATA:context}","msg":"%{GREEDYDATA:message}

I hope, release come soon. Thanks!

jsoriano · 2021-05-28T14:12:28Z

/test

cihantunali · 2021-05-31T06:18:01Z

Hi there!
I hope it can build. Thank you @jsoriano for your effort.

tetianakravchenko · 2021-05-31T08:56:53Z

Hi @jsoriano, as I see that build is failing mainly with timeout error:

E               beat.beat.TimeoutError: Timeout waiting for 'cond' to be true. Waited 10 seconds.

or

E           Failed: Timeout >90.0s

is there anything I can do here?

jsoriano · 2021-05-31T14:00:00Z

@tetianakravchenko try updating your branch with master, failing tests can be related to outdated files.

jsoriano · 2021-06-02T07:47:52Z

I have updated the branch, let's see if tests pass now 🤞

jsoriano · 2021-06-02T11:40:29Z

/test

cihantunali · 2021-06-03T10:20:29Z

I think there is same error here

E Failed: Timeout >90.0s

jsoriano · 2021-06-07T09:53:26Z

I am not sure why other modules fail, but there seems to be a problem with the change in the MongoDB module. Running the tests locally I see this error in filebeat logs:

2021-06-07T07:17:32.045-0200    ERROR   [publisher_pipeline_output]     pipeline/output.go:154  Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): Connection marked as failed because the onConnect callback failed: 1 error: error loading pipeline for fileset mongodb/log: couldn't load pipeline: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'ACCESS'","       ^---- HERE"],"script":"mongodb.c == 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":21},"suppressed":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'WRITE'","       ^---- HERE"],"script":"mongodb.c == 'WRITE'","lang":"painless","position":{"offset":7,"start":0,"end":20}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c != 'WRITE' && mongodb. ...","       ^---- HERE"],"script":"mongodb.c != 'WRITE' && mongodb.c != 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":32}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.s == 'F' || mongodb.s == ...","       ^---- HERE"],"script":"mongodb.s == 'F' || mongodb.s == 'E'","lang":"painless","position":{"offset":7,"start":0,"end":32}}]}],"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'ACCESS'","       ^---- HERE"],"script":"mongodb.c == 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":21},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"},"suppressed":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'WRITE'","       ^---- HERE"],"script":"mongodb.c == 'WRITE'","lang":"painless","position":{"offset":7,"start":0,"end":20},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c != 'WRITE' && mongodb. ...","       ^---- HERE"],"script":"mongodb.c != 'WRITE' && mongodb.c != 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":32},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.s == 'F' || mongodb.s == ...","       ^---- HERE"],"script":"mongodb.s == 'F' || mongodb.s == 'E'","lang":"painless","position":{"offset":7,"start":0,"end":32},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.s]"}}]},"status":400}. Response body: {"error":{"root_cause":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'ACCESS'","       ^---- HERE"],"script":"mongodb.c == 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":21},"suppressed":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'WRITE'","       ^---- HERE"],"script":"mongodb.c == 'WRITE'","lang":"painless","position":{"offset":7,"start":0,"end":20}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c != 'WRITE' && mongodb. ...","       ^---- HERE"],"script":"mongodb.c != 'WRITE' && mongodb.c != 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":32}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.s == 'F' || mongodb.s == ...","       ^---- HERE"],"script":"mongodb.s == 'F' || mongodb.s == 'E'","lang":"painless","position":{"offset":7,"start":0,"end":32}}]}],"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'ACCESS'","       ^---- HERE"],"script":"mongodb.c == 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":21},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"},"suppressed":[{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c == 'WRITE'","       ^---- HERE"],"script":"mongodb.c == 'WRITE'","lang":"painless","position":{"offset":7,"start":0,"end":20},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.c != 'WRITE' && mongodb. ...","       ^---- HERE"],"script":"mongodb.c != 'WRITE' && mongodb.c != 'ACCESS'","lang":"painless","position":{"offset":7,"start":0,"end":32},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.c]"}},{"type":"script_exception","reason":"compile error","processor_type":"append","script_stack":["mongodb.s == 'F' || mongodb.s == ...","       ^---- HERE"],"script":"mongodb.s == 'F' || mongodb.s == 'E'","lang":"painless","position":{"offset":7,"start":0,"end":32},"caused_by":{"type":"illegal_argument_exception","reason":"cannot resolve symbol [mongodb.s]"}}]},"status":400}

There seems to be a problem with the pipeline.

I can reproduce the error running the following command from the filebeat directory: PYTEST_ADDOPTS="-k test_modules" GENERATE=1 TESTING_FILEBEAT_MODULES=mongodb mage pythonIntegTest.

After running the tests, logs are available in build/system-tests/run/test_modules.Test.test_fileset_file_0_mongodb/output.log.

@tetianakravchenko can you take a look?

tetianakravchenko · 2021-06-07T14:02:34Z

thank you for the hint, I will have a look 👍

…move redundant rename

tetianakravchenko · 2021-06-13T21:13:34Z

Hi @jsoriano. Locally I have the command below running successfully after the latest changes

filebeat directory: PYTEST_ADDOPTS="-k test_modules" GENERATE=1 TESTING_FILEBEAT_MODULES=mongodb mage pythonIntegTest

Could you please trigger tests?

jsoriano · 2021-06-14T15:38:06Z

/test

jsoriano · 2021-06-14T15:40:48Z

Thanks @tetianakravchenko! I can confirm that tests pass now locally for me, let's wait for CI 🤞

jsoriano

@tetianakravchenko sorry, one last thing, could you please add a changelog entry in the CHANGELOG.next.asciidoc file? For the rest it LGTM, thanks!

jsoriano · 2021-06-15T11:38:38Z

/test

mergify · 2021-06-15T11:38:39Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b mongodb-json-format upstream/mongodb-json-format
git merge upstream/master
git push upstream mongodb-json-format

jsoriano · 2021-06-15T11:39:29Z

/test

jsoriano

LGTM, thanks!

cihantunali · 2021-06-16T06:30:45Z

G8 work! When it can be released and which version?

jsoriano · 2021-06-16T07:20:48Z

@cihantunali this will be likely available in Filebeat 7.14.0.

MongoDB 4.4 uses structured JSON format for logging, add support for this format while keeping support for the old one too. (cherry picked from commit 3752526)

jsoriano · 2021-06-16T07:47:22Z

Looking again at this change, I think that we should store the message in the message field, as the pipeline for plaintext does, and also I think that there is no need to keep the original log message as it is fully parsed. I am doing these changes as a follow up in #26338

tetianakravchenko · 2021-06-16T08:14:33Z

@jsoriano my idea was to keep the original log message, as it contains attr field, where might be stored important information, like locks, readConcern, writeConcern, command, etc (all those logs are part of slow logs). As content of this field depends on log type, it doesn't make sense to parse each subfield, but keeping original message with attr (other optional fields are truncated, size, tags) for debugging purpose I think would be helpful.

jsoriano · 2021-06-16T08:52:44Z

@tetianakravchenko good point, ok, I will keep the original message but under the ECS log.original field. I have updated the PR, would you mind to review it?
Maybe in the future we can add something to store these attributes.

MongoDB 4.4 uses structured JSON format for logging, add support for this format while keeping support for the old one too. (cherry picked from commit 3752526) Co-authored-by: Tetiana Kravchenko <tanya.kravchenko.v@gmail.com>

jsoriano · 2021-06-17T09:20:26Z

Change applied to integrations in elastic/integrations#1138.

MongoDB 4.4 uses structured JSON format for logging, add support for this format while keeping support for the old one too.

* master: (25 commits) Fix UBI source URL (elastic#26384) Skip test_rotating_file in osx and windows (elastic#26379) Remove outdated k8s manifests for managed elastic-agent (elastic#26368) Enable agent to send custom headers to kibana/ES (elastic#26275) [Automation] Update elastic stack version to 8.0.0-943ef2c0 for testing (elastic#26354) Make the Syslog input GA (elastic#26293) Move Kerberos FAST config flag to shared kerberos config (elastic#26141) Add k8s cluster identifiers (elastic#26056) Store message from MongoDB json logs in message field (elastic#26338) update threatintel ECS version (elastic#26274) update envoyproxy ECS version (elastic#26277) [Filebeat] [MongoDB] Support MongoDB 4.4 json logs (elastic#24774) Update go-structform to 0.0.9 (elastic#26251) Forward port 7.13.2 changelog to master (elastic#26323) Updated filter expression for filtering 86 artifacts (elastic#26313) Osquerybeat: Align with the rest of the beats, set the ECS version (elastic#26324) [Packetbeat] Add `url.extension` to Packetbeat HTTP events (elastic#25999) Change link to snapshots in README (elastic#26317) Don't include full ES index template in errors (elastic#25743) First refactor of the system module - system/cpu and system/core (elastic#25771) ...

tetianakravchenko added 2 commits March 25, 2021 21:52

mongodb ingest pipeline: add pipelines for json; add logs example

6e6d441

add json log and expected.json; fix mongodb pipeline-json

744e734

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 25, 2021

jsoriano added the Team:Integrations Label for the Integrations team label Mar 29, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2021

jsoriano added the review label Mar 29, 2021

add json and plaintext pipelines to the manifest

9f5d34c

add Compatibility documentation: tested with plaintext and json logs

0126480

jsoriano reviewed Mar 30, 2021

View reviewed changes

filebeat/module/mongodb/_meta/docs.asciidoc Show resolved Hide resolved

run make update

9823ac8

Merge remote-tracking branch 'origin/master' into HEAD

21714f0

fix tests: add fields msg and id, fix fields path for json format, re…

9624560

…move redundant rename

P1llus added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 14, 2021

jsoriano reviewed Jun 15, 2021

View reviewed changes

tetianakravchenko and others added 2 commits June 15, 2021 13:32

add a changelog entry

25d8e36

Move changelog entry to correct section

0769350

Merge branch 'master' into mongodb-json-format

be5576a

jsoriano approved these changes Jun 15, 2021

View reviewed changes

jsoriano added the backport-v7.14.0 Automated backport with mergify label Jun 15, 2021

jsoriano merged commit 3752526 into elastic:master Jun 16, 2021

mergify bot mentioned this pull request Jun 16, 2021

[7.x](backport #24774) [Filebeat] [MongoDB] Support MongoDB 4.4 json logs #26336

Merged

jsoriano mentioned this pull request Jun 16, 2021

Store message from MongoDB json logs in message field #26338

Merged

6 tasks

jsoriano mentioned this pull request Jun 16, 2021

Breaking changes for 8.0 #13911

Closed

19 tasks

jsoriano mentioned this pull request Jun 17, 2021

Add support for json logs in mongodb integration elastic/integrations#1138

Merged

4 tasks

jsoriano removed the needs_integration_sync Changes in this PR need synced to elastic/integrations. label Jun 17, 2021

[Filebeat] [MongoDB] Support MongoDB 4.4 json logs #24774

[Filebeat] [MongoDB] Support MongoDB 4.4 json logs #24774

Conversation

tetianakravchenko commented Mar 25, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Mar 25, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

jsoriano commented Mar 29, 2021

elasticmachine commented Mar 29, 2021

jsoriano commented Mar 30, 2021

tetianakravchenko commented Mar 30, 2021

jsoriano commented Mar 30, 2021

jsoriano commented Mar 30, 2021

cihantunali commented May 28, 2021

jsoriano commented May 28, 2021

cihantunali commented May 31, 2021

tetianakravchenko commented May 31, 2021 • edited Loading

jsoriano commented May 31, 2021

jsoriano commented Jun 2, 2021

jsoriano commented Jun 2, 2021

cihantunali commented Jun 3, 2021

jsoriano commented Jun 7, 2021

tetianakravchenko commented Jun 7, 2021

tetianakravchenko commented Jun 13, 2021

jsoriano commented Jun 14, 2021

jsoriano commented Jun 14, 2021

jsoriano left a comment

Choose a reason for hiding this comment

jsoriano commented Jun 15, 2021

mergify bot commented Jun 15, 2021

jsoriano commented Jun 15, 2021

jsoriano left a comment

Choose a reason for hiding this comment

cihantunali commented Jun 16, 2021

jsoriano commented Jun 16, 2021

jsoriano commented Jun 16, 2021

tetianakravchenko commented Jun 16, 2021

jsoriano commented Jun 16, 2021

jsoriano commented Jun 17, 2021

tetianakravchenko commented Mar 25, 2021 •

edited

Loading

elasticmachine commented Mar 25, 2021 •

edited by jenkins-beats-ci bot

Loading

tetianakravchenko commented May 31, 2021 •

edited

Loading