Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Filebeat module issues found during ECS conversion #9208

Closed
4 of 10 tasks
webmat opened this issue Nov 22, 2018 · 3 comments
Closed
4 of 10 tasks

[WIP] Filebeat module issues found during ECS conversion #9208

webmat opened this issue Nov 22, 2018 · 3 comments
Assignees
Labels
Filebeat Filebeat module Team:Integrations Label for the Integrations team v7.0.0

Comments

@webmat
Copy link
Contributor

webmat commented Nov 22, 2018

Please ignore for now.

This is just a collection of all the small problems we notice could be improved while performing the ECS transition (#8655). Some things are small enough that we can tackle them right in the ECS PRs.

Some things are out of scope, though. Here they are:

Usage of http.response.elapsed_time

This field is not in ECS and should be deprecated in favour of event.duration. Modules using it:

  • kibana

Full version strings vs breakdown fields

  • Ingest Node's UA parser gives us the version number in broken up in 3 fields (major/minor/patch), but no full version string (e.g. 10.14.1). We should populate user_agent.version and user_agent.os.version with full version strings. Modules affected:
    • apache2.access
    • iis.access
    • nginx.access
    • traefik.access

system.syslog

  • Add integration test for the convert_timezone option

system.auth

  • Add integration tests for message formats other than sshd, sudo, groupadd and useradd. Right now it's only testing for those, and no test for the last GROK pattern that just populates the message field.
  • Patterns specific to sshd, sudo, groupadd and useradd
  • After performing field renames on auth messages, I wonder if the resulting events are missing something. For example the first pattern uses only ECS fields, if it wasn't for one last field that hasn't been renamed (system.auth.ssh.method, line 29), I would never know that its ECS event is a message about SSH.

nginx.access

  • Current parsing of access log assumes people are using $http_x_forwarded_for rather than the default $remote_addr for their remote address. This means even though nginx can receive connections from unix sockets (e.g. common when polling for nginx stats), this module doesn't support these log events.

Message in error filesets

E.g. apache.error, nginx.error

  • Currently outputs to message rather than error.message, since all Ingest Node documentation currently suggests outputting Ingest Node error messages at error.message.
    • I'm not 100% convinced about using error.message in any case for userland error messages, by the way. We've been talking about always populating message for the timeline viewer.
@ruflin
Copy link
Member

ruflin commented Nov 22, 2018

Thanks for this issue, let's keep adding things. I think if possible we should resolve these issues for 7.0.

@ruflin ruflin added the Team:Integrations Label for the Integrations team label Nov 27, 2018
@alvarolobato alvarolobato added ecs and removed ecs labels Dec 4, 2018
@alvarolobato
Copy link

@ruflin will break this down into individual issues after all the ECS conversion is processed.

@ruflin
Copy link
Member

ruflin commented Jan 24, 2019

Closing this issue.

@ruflin ruflin closed this as completed Jan 24, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Filebeat Filebeat module Team:Integrations Label for the Integrations team v7.0.0
Projects
None yet
Development

No branches or pull requests

3 participants