Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #24699 to 7.x: [Filebeat] Add URI Parts Processor to multiple modules #25353

Merged
merged 1 commit into from
Apr 27, 2021

Conversation

andrewstucki
Copy link
Contributor

@andrewstucki andrewstucki commented Apr 27, 2021

Cherry-pick of PR #24699 to 7.x branch. Original message:

What does this PR do?

Updates Ingest Pipelines for the below modules:

Apache, Nginx, IIS, Traefik, S3Access, Cisco, F5, Fortinet, Google Workspace, Imperva, Microsoft, Netscout, O365, Sophos, Squid, Suricata, Zeek, Zia, Zoom, ZScaler

With the below changes

  • Add uri_parts processor to parse URIs (includes URL decoding) to add url.path, url.extension, url.query....
  • URL Decodes http.request.referrer (when applicable) to make them human readable

Why is it important?

Parses URLs to break up the URL into the different parts and URL decodes them.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

How to test this PR locally

cd beats/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=apache,nginx,iis,traefik mage -v pythonIntegTest
cd beats/x-pack/filebeat
GENERATE=true TESTING_FILEBEAT_MODULES=s3access,cisco,f5,fortinet,google_workspace,imperva,microsoft,netscout,o365,sophos,squid,suricata,zeek,zia,zoom,zscaler mage -v pythonIntegTest

Related issues

Use cases

Screenshots

Logs

* Update Nginx pipelines

* Update Apache, Nginx, IIS, Traefik pipelines

* Update AWS S3

* Update Cisco

* Update F5

* Update Fortinet

* Update Imperva, Netscout, O365, Sophos, Squid, Suricata, Zscaler

* additional fixes

* update pipelines

* unescape \

* remove urldecodes for url.original

* updates after rebase

* update zeek SIP

* update changelog as requested by @andrewstucki

* remove `url_decode` for `http.request.referrer`

* update generated data

(cherry picked from commit f1fea95)
@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 27, 2021
@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #25353 opened

  • Start Time: 2021-04-27T19:49:29.059+0000

  • Duration: 68 min 27 sec

  • Commit: 0c95fd8

Test stats 🧪

Test Results
Failed 0
Passed 13672
Skipped 2300
Total 15972

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13672
Skipped 2300
Total 15972

@andrewstucki andrewstucki merged commit af3d974 into elastic:7.x Apr 27, 2021
@andrewstucki andrewstucki deleted the backport_24699_7.x branch April 27, 2021 21:27
@zube zube bot removed the [zube]: Done label Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants