Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lowercase causes missed detections and broken searches #18154

Closed
neu5ron opened this issue May 2, 2020 · 4 comments · Fixed by #18359
Closed

lowercase causes missed detections and broken searches #18154

neu5ron opened this issue May 2, 2020 · 4 comments · Fixed by #18359
Assignees
@leehinman
Labels
ecs

Comments

@neu5ron
Copy link

neu5ron commented May 2, 2020 •
edited
Loading

In the Zeek HTTP file, there is a lowercasing of http.request.method - shown here:

beats/x-pack/filebeat/module/zeek/http/ingest/pipeline.yml

Line 53 in f39d869

- lowercase:

http.request.method will have values of POST, GET, OPTIONS, etc... the vast majority of dashboards, visualizations, searches, threat hunts, etc.. are all built on how the vast majority of HTTP requests work ie: the value of GET for outbound stuff or POST for inbound web attacks.
couple this with values being case sensitive, there is no even "fail safe" that would have made this not such an impactful thing.
also, this field can be used for anomalous variations of the above, such as looking for PoST.
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/siem (Team:SIEM)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 2, 2020
@andrewkroh andrewkroh added the ecs label May 2, 2020
@leehinman
Copy link
Contributor

I definitely should have marked this as a breaking change, I'll fix that.

The reason for making it lowercase is because it is an ECS field and the doc for the field is:

HTTP request method.

The field value must be normalized to lowercase for querying.
See the documentation section "Implementing ECS".

type: keyword

example: get, post, put

@webmat & @MikePaquette thoughts?

What do you think about having "zeek.http.method" as the unmodified request method?

@neu5ron
Copy link
Author

neu5ron commented May 5, 2020

cross reference for Suricata too, same scenario -

beats/x-pack/filebeat/module/suricata/eve/ingest/pipeline.yml

Line 5 in 6d7d919

- lowercase:

@leehinman
Copy link
Contributor

And Apache & nginx

@leehinman leehinman self-assigned this May 7, 2020
@leehinman leehinman mentioned this issue May 7, 2020
Merged
3 tasks
leehinman added a commit to leehinman/beats that referenced this issue May 7, 2020
@leehinman
Preserve case of http.request.method
566cdbe 
ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes elastic#18154
leehinman added a commit that referenced this issue May 27, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (#18359)
87c3ad3 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes #18154
@leehinman leehinman mentioned this issue May 27, 2020
Merged
3 tasks
leehinman added a commit to leehinman/beats that referenced this issue May 27, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (elastic#18359)
cb6b5f4 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes elastic#18154

(cherry picked from commit 87c3ad3)
@leehinman leehinman mentioned this issue May 27, 2020
Merged
3 tasks
leehinman added a commit to leehinman/beats that referenced this issue May 27, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (elastic#18359)
3414094 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes elastic#18154

(cherry picked from commit 87c3ad3)
@leehinman leehinman mentioned this issue May 27, 2020
Merged
3 tasks
leehinman added a commit to leehinman/beats that referenced this issue May 27, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (elastic#18359)
8f906ad 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes elastic#18154

(cherry picked from commit 87c3ad3)
leehinman added a commit that referenced this issue May 28, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (#18359) (#18782)
c4f3c01 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes #18154

(cherry picked from commit 87c3ad3)
leehinman added a commit that referenced this issue May 28, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (#18359) (#18788)
932b273 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes #18154

(cherry picked from commit 87c3ad3)
leehinman added a commit that referenced this issue May 28, 2020
@leehinman
[Filebeat] Preserve case of http.request.method (#18359) (#18783)
67f6d39 
* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes #18154

(cherry picked from commit 87c3ad3)
leweafan pushed a commit to leweafan/beats that referenced this issue Apr 28, 2023
@leehinman
[Filebeat] Preserve case of http.request.method (elastic#18359) (elas…
99e7c8d 
…tic#18788)

* Preserve case of http.request.method

ECS previously specified normalizing http.request.method to lowercase.
This resulted in the loss of information. Affects filesets from the
following versions:

- apache/access (7.7 - 7.8)
- elasticsearch/audit (7.7 - 7.8)
- iis/access (7.7 - 7.8)
- iis/error (7.7 - 7.8)
- nginx/access (7.8)
- nginx/ingress_controller (7.8)
- aws/elb (7.7 - 7.8)
- suricata/eve (7.4 - 7.8)
- zeek/http (7.8)

Closes elastic#18154

(cherry picked from commit 5490eb4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@leehinman leehinman

Labels
ecs
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Filebeat] Preserve case of http.request.method leehinman/beats
5 participants
@andresrc @andrewkroh @neu5ron @elasticmachine @leehinman