[Filebeat][Cisco ASA] log enhancement and performance #24744

andrewkroh · 2021-03-24T18:39:15Z

The revives #20831 which I didn't have edit permissions on.

What does this PR do?

This PR resolve some reported issues with ECS and Cisco ASA/FTD and is adding new message patterns.

Overview of the Changes:

Adding 15 new message pattern with dissect processor

Fix parsing error not extracting event.outcome and network.transport from 106015
All other processors associated with the new message id's have been updated and extended
Further additional fields were derived from the logs
Changed event.outcome in script processor to ECS defined values.
Adding anchors to grok patterns with no conditional and to grok processors that using more than one pattern
Adding new event.action for user creation/deletion or bypass events
Fix 106014

Why is it important?

We think that these are one of the most used message types in Cisco ASA logs.
Adding the anchors increases the throughput/performance. It is described in more detail in this blog article https://www.elastic.co/blog/do-you-grok-grok. We need more event.actions for specific logs/events.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.
I have made corresponding changes to the documentation

Related issues

- Fixed some ECS issues - added anchors on grok patterns for performance - added messages: ------------------------- 434004 434002 713905 750002 750003 110002 419002 602304 602303 713120 713202 713901 713904 713906 713905 ------------------------- - with the messages pattern added also this commit add four new event action types in the script that mapped event actions to the event.kind/category/type - added set processor for adding outcome, action and protocol if necessary for the new messages

fix parsing error and add enhancements

fix 602303

commit after running tests.

… space in between is optional in log message

This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case.

…ePattern

…/beats into evoila-ingestCiscoMessagePattern

elasticmachine · 2021-03-24T18:48:11Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

elasticmachine · 2021-03-24T18:59:11Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24744 updated
Start Time: 2021-04-19T21:46:15.294+0000
Duration: 57 min 6 sec
Commit: 0b32165

Test stats 🧪

Test	Results
Failed	0
Passed	6928
Skipped	1185
Total	8113

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	6928
Skipped	1185
Total	8113

andrewkroh · 2021-03-24T19:00:09Z

jenkins, run tests

mergify · 2021-04-07T16:27:44Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b evoila-ingestCiscoMessagePattern upstream/evoila-ingestCiscoMessagePattern
git merge upstream/master
git push upstream evoila-ingestCiscoMessagePattern

adriansr

LGTM

…MessagePattern

* ecs fix - more message pattern - Fixed some ECS issues - added anchors on grok patterns for performance - added messages: ------------------------- 434004 434002 713905 750002 750003 110002 419002 602304 602303 713120 713202 713901 713904 713906 713905 ------------------------- - with the messages pattern added also this commit add four new event action types in the script that mapped event actions to the event.kind/category/type - added set processor for adding outcome, action and protocol if necessary for the new messages * Update asa-ftd-pipeline.yml * Update asa-ftd-pipeline.yml fix parsing error and add enhancements * Update asa-ftd-pipeline.yml fix 602303 * testing for PR and some minor fixes * commit for requested changes * newline * test * make test commit commit after running tests. * Fix parsing on 106014 with an additional ${SPACE} in grok pattern, so space in between is optional in log message * fixed 106014 finally This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case. * after test commit * Test after merge * Update generated * Add changelog * Undo meraki generated file changes * Update generated Co-authored-by: pcosic <pcosic@evoila.de> Co-authored-by: pcosic <69909732+pcosic@users.noreply.github.com> (cherry picked from commit 226485b)

…-github-pr-comment-template * upstream/master: [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) Add svc to agent k8s clusterRole (elastic#25146) Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041) [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744) Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117) Cyberark Privileged Access Security module (elastic#24803) [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150) Fix for tests after `device...` field has been removed (elastic#25141) [Ingest Manager] Restart process on output change (elastic#24907) Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137) [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047) Add support for ignore_inactive in filestream input (elastic#25036) Fix bug with annotations dedot config on k8s not used (elastic#25111)

* ecs fix - more message pattern - Fixed some ECS issues - added anchors on grok patterns for performance - added messages: ------------------------- 434004 434002 713905 750002 750003 110002 419002 602304 602303 713120 713202 713901 713904 713906 713905 ------------------------- - with the messages pattern added also this commit add four new event action types in the script that mapped event actions to the event.kind/category/type - added set processor for adding outcome, action and protocol if necessary for the new messages * Update asa-ftd-pipeline.yml * Update asa-ftd-pipeline.yml fix parsing error and add enhancements * Update asa-ftd-pipeline.yml fix 602303 * testing for PR and some minor fixes * commit for requested changes * newline * test * make test commit commit after running tests. * Fix parsing on 106014 with an additional ${SPACE} in grok pattern, so space in between is optional in log message * fixed 106014 finally This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case. * after test commit * Test after merge * Update generated * Add changelog * Undo meraki generated file changes * Update generated Co-authored-by: pcosic <pcosic@evoila.de> Co-authored-by: pcosic <69909732+pcosic@users.noreply.github.com> (cherry picked from commit 226485b)

… (#25158) * [Filebeat][Cisco ASA] log enhancement and performance (#24744) * ecs fix - more message pattern - Fixed some ECS issues - added anchors on grok patterns for performance - added messages: ------------------------- 434004 434002 713905 750002 750003 110002 419002 602304 602303 713120 713202 713901 713904 713906 713905 ------------------------- - with the messages pattern added also this commit add four new event action types in the script that mapped event actions to the event.kind/category/type - added set processor for adding outcome, action and protocol if necessary for the new messages * Update asa-ftd-pipeline.yml * Update asa-ftd-pipeline.yml fix parsing error and add enhancements * Update asa-ftd-pipeline.yml fix 602303 * testing for PR and some minor fixes * commit for requested changes * newline * test * make test commit commit after running tests. * Fix parsing on 106014 with an additional ${SPACE} in grok pattern, so space in between is optional in log message * fixed 106014 finally This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case. * after test commit * Test after merge * Update generated * Add changelog * Undo meraki generated file changes * Update generated Co-authored-by: pcosic <pcosic@evoila.de> Co-authored-by: pcosic <69909732+pcosic@users.noreply.github.com> (cherry picked from commit 226485b) * geoip updates Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

pcosic and others added 18 commits August 27, 2020 16:56

Update asa-ftd-pipeline.yml

8e0491e

Update asa-ftd-pipeline.yml

b13af6e

fix parsing error and add enhancements

Update asa-ftd-pipeline.yml

eafaae2

fix 602303

testing for PR and some minor fixes

40814cc

commit for requested changes

b992fcd

newline

5386064

test

1775792

Merge branch 'master' into ingestCiscoMessagePattern

14aebc0

make test commit

029083f

commit after running tests.

Fix parsing on 106014 with an additional ${SPACE} in grok pattern, so…

e398834

… space in between is optional in log message

fixed 106014 finally

1e9da38

This fixing finally 106014. We have, afaik, two options. Use IPORHOST to not match '(type' or using '(?<destination.address>[^ (]*)' so we only dispense on space or '(' for the case destination.address is weird. NOTSPACE is not work in this case.

Merge remote-tracking branch 'upstream/master' into ingestCiscoMessag…

e664cd6

…ePattern

after test commit

babe7b5

Merge remote-tracking branch 'upstream/master' into ingestCiscoMessag…

faf2659

…ePattern

Test after merge

763132e

Merge branch 'ingestCiscoMessagePattern' of https://github.com/evoila…

195e645

…/beats into evoila-ingestCiscoMessagePattern

Update generated

24aef0f

andrewkroh added the Team:Security-External Integrations label Mar 24, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Mar 24, 2021

Add changelog

0abf355

andrewkroh added bug enhancement Filebeat Filebeat labels Mar 24, 2021

andrewkroh marked this pull request as ready for review March 24, 2021 18:48

andrewkroh mentioned this pull request Mar 24, 2021

[Filebeat][Cisco ASA] log enhancement and performance #20831

Closed

6 tasks

Undo meraki generated file changes

9b154e4

adriansr approved these changes Apr 19, 2021

View reviewed changes

andrewkroh added 3 commits April 19, 2021 09:09

Merge remote-tracking branch 'elastic/master' into evoila-ingestCisco…

1bf67a2

…MessagePattern

Merge remote-tracking branch 'elastic/master' into evoila-ingestCisco…

a5d8c88

…MessagePattern

Update generated

0b32165

andrewkroh merged commit 226485b into elastic:master Apr 19, 2021

andrewkroh added backport-v7.13.0 Automated backport with mergify needs_integration_sync Changes in this PR need synced to elastic/integrations. labels Apr 19, 2021

mergify bot mentioned this pull request Apr 19, 2021

[Filebeat][Cisco ASA] log enhancement and performance (backport #24744) #25158

Merged

adriansr mentioned this pull request Nov 3, 2021

Cisco package Needs sync with Beats module elastic/integrations#2116

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat][Cisco ASA] log enhancement and performance #24744

[Filebeat][Cisco ASA] log enhancement and performance #24744

andrewkroh commented Mar 24, 2021 •

edited

Loading

elasticmachine commented Mar 24, 2021

elasticmachine commented Mar 24, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

andrewkroh commented Mar 24, 2021

mergify bot commented Apr 7, 2021

adriansr left a comment

[Filebeat][Cisco ASA] log enhancement and performance #24744

[Filebeat][Cisco ASA] log enhancement and performance #24744

Conversation

andrewkroh commented Mar 24, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Related issues

elasticmachine commented Mar 24, 2021

elasticmachine commented Mar 24, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

andrewkroh commented Mar 24, 2021

mergify bot commented Apr 7, 2021

adriansr left a comment

Choose a reason for hiding this comment

andrewkroh commented Mar 24, 2021 •

edited

Loading

elasticmachine commented Mar 24, 2021 •

edited by jenkins-beats-ci bot

Loading