Convert apache2.access to ECS - Take 2 #9245
Conversation
|
@ruflin I brought your initial Apache2 access PR exactly in line with the other recent access log PRs. Expecting to have the build work the first time. |
webmat
force-pushed
the
ecs-apache2-access
branch
from
53da6b2
to
5d95880
Compare
webmat
force-pushed
the
ecs-apache2-access
branch
from
5d95880
to
321c5a9
Compare
|
Only failure in Jenkins was a network hiccup on Ubuntu:
|
* Update ecs-migration.yml file * Update changelog * Update generated files * Link old fields Todo: * Add aliases for old fields
- remove `source.hostname` field definition - remove misspelled `http.request.referer`. The rreal one is already defined. - pipeline reverted to populate `apache2.access.remote_ip`. - add split to source.ip or source.domain as a separate grok.
- Output the ua string to the field not afflicted with a typo. - Fix user agent parsing - Add a log entry with a hostname as the remote host, instead of an IP - Add leniency to geoip filter, if remote host is a hostname instead of an IP (no ip field)
- Remove unneeded definition for `http.response.body_sent.bytes`. - Re-introduce `apache2.access.remote_ip` field definition. - Alias all `apache2.access.*` that have moved.
Dupe of definition in `libbeat/_meta/fields.ecs.yml`.
webmat
force-pushed
the
ecs-apache2-access
branch
from
f5ddc46
to
54a294e
Compare
|
@ruflin Ok, ready for final review. Everything is green, and even fixes the test breakage in master. |
|
For a future change, we might consider to rename the |
|
@jsoriano Actually "Apache" is the foundation. The webserver is actually called "httpd" ;-) |
|
But I totally agree on another part of your point. The number is problematic. Will "apache2" support Apache httpd v3? So yeah, the number has to go eventually |
|
@webmat apart of the number I was mentioning that to use the same name as the metricbeat module, though we can also rename both to |
|
Ah I was not aware of that. Thanks for mentioning it! |
- Convert many fields under `apache2.access.*` to ECS. Previous field names are field aliases towards the new corresponding ECS field: - apache2.access.user_name => user.name - apache2.access.method => http.request.method - apache2.access.url => url.original - apache2.access.http_version => http.version - apache2.access.response_code => http.response.status_code - apache2.access.referrer => http.request.referrer - apache2.access.agent => user_agent.original - read_timestamp => event.created - apache2.access.geoip.* => source.geo.* - apache2.access.user_agent.* => user_agent.* - Rename common field `url.href` to `url.original`, bringing it in line with ECS - Remove field defs which are no longer in ECS: - url.href.raw - source.hostname - Add a log entry with a hostname as the remote host, instead of an IP - Add leniency to geoip filter, if remote host is a hostname instead of an IP (no ip field) - Coerce int fields to ints: status_code, sent bytes - Fix unrelated test failure caused by generated code (under x-pack/metricbeat) that seems to be outdated in master
Taking over #8901, and I don't have write access to @ruflin's repo.
TODO:
url.originalfromfilebeat/_meta/fields.common.yml. It's a dupe oflibbeat/_meta/fields.ecs.ymldefinition.https://travis-ci.org/elastic/beats/jobs/460918667Unrelated
Closes #8901, partially addresses #8655.