Convert Filebeat nginx.access to ECS #9081

webmat · 2018-11-14T20:22:42Z

Caveat

This module assumes that the nginx log format is a $http_x_forwarded_for, which is either one IP or more. This means any log line that would contain a unix socket (one possibility, with nginx' default $remote_addr) is already not parsed successfully by this module. This field mapping to ECS does not introduce support for unix sockets.

Renames

nginx.access.remote_ip_list => network.forwarded_ip
nginx.access.remote_ip => source.ip
nginx.access.user_name => user.name
nginx.access.response_code => http.response.status_code
nginx.access.referrer => http.request.referrer
nginx.access.method => http.request.method
nginx.access.url => url.original
nginx.access.http_version => http.version
nginx.access.agent => user_agent.original
nginx.access.user_agent.* => user_agent.*
nginx.access.geoip.* => source.geo.*

TODO

Key 'http.request.referrer' found in event is not documented!
Update ECS-migration.yml file
Changelog
Output user_agent to ECS field names
Simplify the code, based on realization that doesn't support unix sockets in the remote address
Create field aliases
Fix test failure Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 400 Bad Request: {"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Invalid [path] value [http.referrer] for field alias [nginx.access.referrer]: an alias must refer to an existing field in the mappings."}]

ruflin

Could you add 2 things to the todo list:

Update Changelog
Update ECS-migration.yml file

filebeat/module/nginx/access/ingest/default.json

filebeat/module/nginx/access/test/test.log-expected.json

CHANGELOG.asciidoc

webmat · 2018-11-20T13:49:35Z

@ruflin Ready for review

ruflin · 2018-11-21T06:36:12Z

I looked into the user_agent problem. The issue is related to the mapping. The user_agent processor tries to create the field "user_agent.os": "Mac OS X 10.12", . But user_agent.os is actually defined as an object which I think is correct. What we need to do is after the user_agent processor run, we need to modify / rename the os field. This is probably the case for all other places where we use the user_agent processor.

webmat · 2018-11-21T18:39:43Z

Ok, thanks for figuring this out. I'll go around your feedback on all other PRs. Then I think I'll fix user_agent in each of them before merging. I'll also create the aliases in each module's field defs in each PR.

webmat · 2018-11-23T16:53:28Z

@ruflin Ready for final review.

New spin on nginx' IP/socket duality. Since this module groks for an IP list, it means that it doesn't currently support log events with a Unix socket as the source. Fixing this is out of scope for this ECS translation, so I've added a note to #9208 to that effect.

We have exactly one custom field left: nginx.access.body_sent.bytes.

ruflin · 2018-11-23T18:02:41Z

filebeat/module/nginx/access/_meta/fields.yml

    - name: remote_ip
-      type: keyword
-      description: >
-        Client IP address. The first public IP address from the `remote_ip_list` array. If no public IP


I wonder if at some stage we regret removing these descriptions. Ok for now, it's also in git.

Can alias fields have a description? If so, I don't mind restoring the definitions.

There's a bunch of work needing to happen there anyway for doc generation, right? At this time, these aliases get the module's field name replaced with the ECS field name instead.

So if you want to keep the old field descriptions there, we'll need to have a doc generator that

lists the old name

mentions that it's now an alias to the new field, with a mention of the new field (even a link to its definition)

if a description is provided, provide the description of the deprecated field

ruflin · 2018-11-23T18:04:35Z

It could be that CI failure is related. Let's see what Jenkins thinks.

webmat · 2018-11-23T18:24:49Z

Re-kicked both failing Travis jobs. They both died after a 10m timeout.

webmat · 2018-11-23T18:28:36Z

Jenkins failures only on Windows, for these 3 jobs (test report):

test_crawler.Test.test_file_disappear_appear
test_harvester.Test.test_close_removed
test_registrar.Test.test_clean_removed_with_clean_inactive

This has nothing to do with the nginx module. Probably flaky tests?

ruflin · 2018-11-23T18:32:15Z

@webmat Yes the 3 you mention above are flaky tests and I remember @ph wanted to skip them, so potentially already skipped in master (need to catch on PR's from today).

For the failing travis jobs: The part I worry is that it's both in Filebeat. Often when module tests time out it means one is really slow or hanging. But if Jenkins goes green, all should be good. Probably worth restarting the travis ones.

webmat · 2018-11-23T18:39:12Z

One of the Travis jobs is pretty far along in running the tests, this time, while the other one hangs really early: https://travis-ci.org/elastic/beats/jobs/458876579

ph · 2018-11-23T18:57:16Z

@ruflin @jsoriano beat me to it in #9217 not merged yet. I added the issue to the our sync next week.

…roxy IPs

- nginx.access.referrer => http.request.referrer - nginx.access.method => http.request.method - nginx.access.url => url.original - nginx.access.http_version => http.version

- Copy remote_ip to `source.ip` if it's an IP - Copy remote_ip to `source.domain` otherwise - Perform geoIP on `source.ip` when present - Output geo information to `source.geo` - nginx.access.geoip.* => source.geo.*

- `http.request.referrer` - `url.original` This is strange. Not sure how I pushed this PR with these failures unsolved...

Adding a dumb `set` for source.ip, otherwise in the painless context, `source` itself doesn't exist yet. And adding this dictionary in painless sounded more painful than using `set`.

webmat · 2018-11-26T21:23:43Z

@ruflin Figured out why stuff was always hanging: was incorrectly aliasing to http.referrer, rather than http.request.referrer.

Looks like all tests will be green here as well. Will merge tomorrow unless something comes up.

ruflin mentioned this pull request Nov 14, 2018

Transition Beats to ECS #8655

Closed

webmat requested a review from ruflin November 14, 2018 20:24

webmat self-assigned this Nov 14, 2018

webmat added in progress Pull request is currently in progress. module Filebeat Filebeat ecs labels Nov 14, 2018

ruflin reviewed Nov 19, 2018

View reviewed changes

filebeat/module/nginx/access/ingest/default.json Outdated Show resolved Hide resolved

filebeat/module/nginx/access/ingest/default.json Outdated Show resolved Hide resolved

filebeat/module/nginx/access/test/test.log-expected.json Outdated Show resolved Hide resolved

webmat force-pushed the ecs-nginx-access branch from 85ab10d to 47be6ac Compare November 20, 2018 03:04

webmat removed the in progress Pull request is currently in progress. label Nov 20, 2018

ruflin reviewed Nov 20, 2018

View reviewed changes

CHANGELOG.asciidoc Show resolved Hide resolved

webmat added the review label Nov 20, 2018

webmat changed the title ~~[WIP] Convert Filebeat nginx.access to ECS~~ Convert Filebeat nginx.access to ECS Nov 20, 2018

webmat force-pushed the ecs-nginx-access branch 2 times, most recently from 35bc4ed to c971d88 Compare November 23, 2018 16:38

ruflin approved these changes Nov 23, 2018

View reviewed changes

webmat force-pushed the ecs-nginx-access branch from c971d88 to ef89b6d Compare November 23, 2018 21:42

Mathieu Martin added 3 commits November 26, 2018 13:11

Use network.forwarded_ip instead of nginx.access.remote_ip_list for p…

8dda0e6

…roxy IPs

nginx.access.user_name => user.name

06f0f00

nginx.access.response_code => http.response.status_code

33fa659

Mathieu Martin added 12 commits November 26, 2018 13:11

Rename 3 http fields and url:

f010533

- nginx.access.referrer => http.request.referrer - nginx.access.method => http.request.method - nginx.access.url => url.original - nginx.access.http_version => http.version

nginx.access.agent => user_agent.original

6eb3473

Interpret remote host:

c35d745

- Copy remote_ip to `source.ip` if it's an IP - Copy remote_ip to `source.domain` otherwise - Perform geoIP on `source.ip` when present - Output geo information to `source.geo` - nginx.access.geoip.* => source.geo.*

Add two temporary definitions:

7298b75

- `http.request.referrer` - `url.original` This is strange. Not sure how I pushed this PR with these failures unsolved...

Coerce status code and bytes into ints

afb743d

Shove all the renames so far in ecs-migration.yml

3e15592

Changelog

4957185

Perform the user_agent dance.

76278fc

Replace one of 3 Mac/FF user agent with Mac/Chrome, to spice things up

70ba669

Remove grok to detect non-IPs in source.ip...

0d25bbe

Adding a dumb `set` for source.ip, otherwise in the painless context, `source` itself doesn't exist yet. And adding this dictionary in painless sounded more painful than using `set`.

Alias all the fields in the fileset

00cc78e

Move the last field not aliased to the top, to promote sanity

72a22e4

webmat force-pushed the ecs-nginx-access branch from ef89b6d to 72a22e4 Compare November 26, 2018 18:11

Fix alias to http.request.referrer, not http.referrer

bb4c680

ruflin approved these changes Nov 27, 2018

View reviewed changes

webmat merged commit 0edd07a into elastic:master Nov 27, 2018

webmat deleted the ecs-nginx-access branch November 27, 2018 15:47

ruflin mentioned this pull request Dec 4, 2018

Logging UI ECS Migration elastic/kibana#26619

Closed

3 tasks

webmat mentioned this pull request Dec 6, 2018

Re-introduce a simplified version of user_agent. elastic/ecs#240

Merged

ruflin mentioned this pull request Dec 11, 2018

Introduce aliases for 7.x fields in 6.x #9283

Merged

webmat mentioned this pull request Dec 17, 2018

Fix recently translated Filebeat modules where integer coercions were added #9598

Closed

14 tasks

ruflin mentioned this pull request Dec 18, 2018

Logging UI ECS Migration elastic/kibana#27284

Closed

18 tasks

webmat mentioned this pull request Jan 23, 2019

Undo field migration of nginx.access.remote_ip_list. #10272

Merged

1 task

Randy-312 mentioned this pull request Sep 10, 2019

ECS - Squid Log mapping elastic/ecs#549

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Filebeat nginx.access to ECS #9081

Convert Filebeat nginx.access to ECS #9081

webmat commented Nov 14, 2018 •

edited

Loading

ruflin left a comment

webmat commented Nov 20, 2018

ruflin commented Nov 21, 2018

webmat commented Nov 21, 2018

webmat commented Nov 23, 2018

ruflin Nov 23, 2018

webmat Nov 23, 2018

ruflin commented Nov 23, 2018

webmat commented Nov 23, 2018

webmat commented Nov 23, 2018

ruflin commented Nov 23, 2018

webmat commented Nov 23, 2018

ph commented Nov 23, 2018

webmat commented Nov 26, 2018

Convert Filebeat nginx.access to ECS #9081

Convert Filebeat nginx.access to ECS #9081

Conversation

webmat commented Nov 14, 2018 • edited Loading

ruflin left a comment

Choose a reason for hiding this comment

webmat commented Nov 20, 2018

ruflin commented Nov 21, 2018

webmat commented Nov 21, 2018

webmat commented Nov 23, 2018

ruflin Nov 23, 2018

Choose a reason for hiding this comment

webmat Nov 23, 2018

Choose a reason for hiding this comment

ruflin commented Nov 23, 2018

webmat commented Nov 23, 2018

webmat commented Nov 23, 2018

ruflin commented Nov 23, 2018

webmat commented Nov 23, 2018

ph commented Nov 23, 2018

webmat commented Nov 26, 2018

webmat commented Nov 14, 2018 •

edited

Loading