[Filebeat] Update Fortinet Ingest Pipeline #24816

legoguy1000 · 2021-03-29T14:03:56Z

What does this PR do?

Updates the Ingest pipeline for the Fortinet firewall module.

Consolidated some of the duplicate processors across the sub-pipelines
Add uri_parts , user_agent, community_id processors
Set observer.serial_number
Added event.kind: alert for certain UTM events

BREAKING CHANGE:

Removed the rename of fortinet.firewall.eventtype -> event.action and instead set event.action to fortinet.firewall.action for the UTM events to match the other events.

Why is it important?

Added additional log samples, updated certain fields, removed duplicate actions.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine · 2021-03-29T14:17:42Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: P1llus commented: /test
Start Time: 2021-06-29T12:37:38.776+0000
Duration: 109 min 30 sec
Commit: f026be2

Test stats 🧪

Test	Results
Failed	0
Passed	14231
Skipped	2311
Total	16542

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	14231
Skipped	2311
Total	16542

legoguy1000 · 2021-03-30T05:07:43Z

Still working this. Need a 2nd opinion on the 3rd change in the description. Make sure that people think thats the right choice. @ijokarumawak You opened the original issue, what do you think? Also all the documents have event.outcome: success even when the firewall is reporting blocked/denied. Per the ECS spec it should be from the point of view of who generated the event (aka the client) so anything that is denied should have event.outcome: failure . I plan to update that tomorrow. If anyone thinks that wrong, let me know. Then i will take PR out of draft.

ijokarumawak

Hi @legoguy1000 . Thanks for preparing PR! I've looked at the change and found what I requested. So, I am +1 on that aspect. However, I found some not user friendly Kibana UI behavior in Security app. Please check my review comments.

x-pack/filebeat/module/fortinet/firewall/ingest/pipeline.yml

x-pack/filebeat/module/fortinet/firewall/ingest/utm.yml

x-pack/filebeat/module/fortinet/firewall/test/utm.log

legoguy1000 · 2021-04-01T02:45:49Z

@ijokarumawak did you have any concerns about

Removed the rename of fortinet.firewall.eventtype -> event.action and instead set event.action to fortinet.firewall.action for the UTM events to match the other events.

or the event.outcome changes. If not then I will take this PR out of draft.

ijokarumawak · 2021-04-01T03:13:28Z

@legoguy1000 I personally don't have any concern about that. I recommend to go ahead and let others review this PR, too.

elasticmachine · 2021-04-01T03:16:31Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-04-19T19:35:48Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 22136-fortinet-module upstream/22136-fortinet-module
git merge upstream/master
git push upstream 22136-fortinet-module

mergify · 2021-04-28T15:58:28Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 22136-fortinet-module upstream/22136-fortinet-module
git merge upstream/master
git push upstream 22136-fortinet-module

mergify · 2021-05-03T14:02:47Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 22136-fortinet-module upstream/22136-fortinet-module
git merge upstream/master
git push upstream 22136-fortinet-module

mergify · 2021-05-06T08:47:34Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 22136-fortinet-module upstream/22136-fortinet-module
git merge upstream/master
git push upstream 22136-fortinet-module

mergify · 2021-06-28T09:35:22Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 22136-fortinet-module upstream/22136-fortinet-module
git merge upstream/master
git push upstream 22136-fortinet-module

P1llus · 2021-06-29T12:37:25Z

/test

P1llus

Small comment, the rest looks good!

x-pack/filebeat/module/fortinet/firewall/ingest/traffic.yml

* 22136: Update Fortinet Ingest Pipeline * Update Pipelines * Additional updates * Set virus/ips subtypes to event.kind: alert * update fields * Consolidate processors to script * Update event.outcome logic * replace hashmap * update event.outcome * cleanup * Added Changes for #25254 * regenerate data * update changelog * remove extra items in changelog (cherry picked from commit 890e473)

…arwin-arm64 * upstream/master: (295 commits) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) [Metricbeat] Add Couchbase's Sync Gateway module (elastic#25599) Refactor add_cloud_metadata to handle ECS fields easier (elastic#26438) [Elastic Agent] Improper casting of int64 (elastic#26520) [Elastic Agent] Enable configuring monitoring namespace (elastic#26439) [Heartbeat] configure permissions for synthetics config (elastic#26393) Osquerybeat: set the raw index name to supress the timestamp suffix (elastic#26545) [Heartbeat] add screenshots config to synthetics (elastic#26455) [Elastic Agent] Use http2 to connect to Fleet Server. (elastic#26474) Remove all docs about Beats central management (elastic#26399) update data.json for gcp billing (elastic#26506) Skip x-pack metricbeat tests (elastic#26537) [Elastic Agent] Fix issue with FLEET_CA not being used with Fleet Server in container (elastic#26529) Add changelog entry for elastic#26224 (elastic#26531) ...

* 22136: Update Fortinet Ingest Pipeline * Update Pipelines * Additional updates * Set virus/ips subtypes to event.kind: alert * update fields * Consolidate processors to script * Update event.outcome logic * replace hashmap * update event.outcome * cleanup * Added Changes for #25254 * regenerate data * update changelog * remove extra items in changelog (cherry picked from commit 890e473) Co-authored-by: Alex Resnick <adr8292@gmail.com>

* master: (25 commits) fix: Force PLATFORMS environment variable when we build Elastic Agent dependencies on arm64 (elastic#26415) macos for metricbeat to run in the extended meta-stage (elastic#26573) Packaging: add arm7 platform in the main pipeline (elastic#26575) [Heartbeat] Skip flakey timer queue test (elastic#26592) Update to "read_pipeline" permission (elastic#26465) (elastic#26580) API keys do not reflect the need for read_pipeline (elastic#26466) (elastic#26582) Add Fleet agent.id to Agent monitoring data (elastic#26548) Add kinesis metricset (elastic#25989) Refactor of system/memory metricset (elastic#26334) Introduce httpcommon package in libbeat (add support for Proxy) (elastic#25219) [Filebeat] change multiline configuration in awss3 input to parsers (elastic#25873) docs: Hint for the error "Error extracting container id" (elastic#25824) [Docs] Fixed metricbeat redis exported field CPU descriptions (elastic#25846) (elastic#26496) Update urllib to 1.26.5. (elastic#26380) Update golang.org/x/crypto (elastic#26448) [Filebeat] Update Fortinet Ingest Pipeline (elastic#24816) Move parsers outside of filestream input so others can use them as well (elastic#26541) [Filebeat] Fix `threatintel.indicator.url.full` field not populating (elastic#26508) [Filebeat] Add network direction processor to Zeek and Suricata modules (elastic#24620) Logging code cleanup related to Nomad auto-discovery (elastic#26498) ...

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2021

jsoriano added the Team:Security-External Integrations label Mar 29, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 29, 2021

legoguy1000 force-pushed the 22136-fortinet-module branch from 306b257 to fd6da1d Compare March 30, 2021 04:34

legoguy1000 mentioned this pull request Mar 30, 2021

[Filebeat][Fortinet] Fortinet ingest pipeline should set event.kind: alert #22136

Closed

legoguy1000 force-pushed the 22136-fortinet-module branch from 45de05f to ce002a4 Compare March 30, 2021 13:09

ijokarumawak reviewed Mar 31, 2021

View reviewed changes

x-pack/filebeat/module/fortinet/firewall/ingest/pipeline.yml Outdated Show resolved Hide resolved

x-pack/filebeat/module/fortinet/firewall/ingest/utm.yml Show resolved Hide resolved

x-pack/filebeat/module/fortinet/firewall/test/utm.log Show resolved Hide resolved

legoguy1000 marked this pull request as ready for review April 1, 2021 03:16

legoguy1000 force-pushed the 22136-fortinet-module branch from 9cb1c97 to 373622e Compare April 2, 2021 11:54

legoguy1000 force-pushed the 22136-fortinet-module branch 3 times, most recently from 7eb85f3 to 84ac38b Compare April 28, 2021 00:42

legoguy1000 mentioned this pull request Apr 28, 2021

[Filebeat] Rename rsa.misc.hardware_id to observer.serial_number for fortinet.fortimail and fortinet.fortimanager modules #25356

Closed

6 tasks

jamiehynds added the enhancement label Apr 28, 2021

legoguy1000 force-pushed the 22136-fortinet-module branch from e25e744 to 3e33f7e Compare April 28, 2021 15:57

legoguy1000 force-pushed the 22136-fortinet-module branch from 3e33f7e to 9ddb914 Compare April 28, 2021 20:40

legoguy1000 force-pushed the 22136-fortinet-module branch from 9ddb914 to 043b0d5 Compare May 3, 2021 14:24

andrewkroh added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label May 3, 2021

andrewkroh self-assigned this May 3, 2021

legoguy1000 force-pushed the 22136-fortinet-module branch 2 times, most recently from 3ddc7e0 to 3ad2b14 Compare May 6, 2021 23:20

legoguy1000 force-pushed the 22136-fortinet-module branch from a2c1f04 to eff72f8 Compare June 25, 2021 11:37

legoguy1000 added 13 commits June 28, 2021 13:09

22136: Update Fortinet Ingest Pipeline

71ee42b

Update Pipelines

8b00b2e

Additional updates

84ac50e

Set virus/ips subtypes to event.kind: alert

77e33d8

update fields

b04eced

Consolidate processors to script

13fe5ec

Update event.outcome logic

bb1a575

replace hashmap

3a792d6

update event.outcome

206de62

cleanup

b379e2e

Added Changes for elastic#25254

a34fd0a

regenerate data

637051c

update changelog

1373276

legoguy1000 force-pushed the 22136-fortinet-module branch from eff72f8 to 1373276 Compare June 28, 2021 13:10

remove extra items in changelog

f026be2

P1llus assigned P1llus and unassigned andrewkroh Jun 29, 2021

P1llus self-requested a review June 29, 2021 12:37

P1llus approved these changes Jun 29, 2021

View reviewed changes

x-pack/filebeat/module/fortinet/firewall/ingest/traffic.yml Show resolved Hide resolved

P1llus merged commit 890e473 into elastic:master Jun 29, 2021

P1llus added the backport-v7.14.0 Automated backport with mergify label Jun 29, 2021

mergify bot mentioned this pull request Jun 29, 2021

[7.x](backport #24816) [Filebeat] Update Fortinet Ingest Pipeline #26576

Merged

andresrc mentioned this pull request Jul 30, 2021

docs: Prepare Changelog for 7.14.0 #27150

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Filebeat] Update Fortinet Ingest Pipeline #24816

[Filebeat] Update Fortinet Ingest Pipeline #24816

legoguy1000 commented Mar 29, 2021 •

edited

Loading

elasticmachine commented Mar 29, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

legoguy1000 commented Mar 30, 2021

ijokarumawak left a comment

legoguy1000 commented Apr 1, 2021

ijokarumawak commented Apr 1, 2021

elasticmachine commented Apr 1, 2021

mergify bot commented Apr 19, 2021

mergify bot commented Apr 28, 2021

mergify bot commented May 3, 2021

mergify bot commented May 6, 2021

mergify bot commented Jun 28, 2021

P1llus commented Jun 29, 2021

P1llus left a comment

[Filebeat] Update Fortinet Ingest Pipeline #24816

[Filebeat] Update Fortinet Ingest Pipeline #24816

Conversation

legoguy1000 commented Mar 29, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

Use cases

Screenshots

Logs

elasticmachine commented Mar 29, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

legoguy1000 commented Mar 30, 2021

ijokarumawak left a comment

Choose a reason for hiding this comment

legoguy1000 commented Apr 1, 2021

ijokarumawak commented Apr 1, 2021

elasticmachine commented Apr 1, 2021

mergify bot commented Apr 19, 2021

mergify bot commented Apr 28, 2021

mergify bot commented May 3, 2021

mergify bot commented May 6, 2021

mergify bot commented Jun 28, 2021

P1llus commented Jun 29, 2021

P1llus left a comment

Choose a reason for hiding this comment

legoguy1000 commented Mar 29, 2021 •

edited

Loading

elasticmachine commented Mar 29, 2021 •

edited by jenkins-beats-ci bot

Loading