Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Finish migration to event.duration for 4 Filebeat modules #10274

Merged
merged 5 commits into from
Jan 24, 2019

Conversation

webmat
Copy link
Contributor

@webmat webmat commented Jan 23, 2019

This PR is built on top of #10188 and should not be merged before it.

Please only review commits from Jan 23rd or later.

Filebeat modules affected

  • elasticsearch.slowlog
  • haproxy.log
  • mysql.slowlog
  • postgresql.log

Introduced in this PR

  • All pipelines use a temp value and a common Ingest Node painless processor, to reduce script compilations.
    • MySQL uses temp field but not shared code, as it requires a tweak to avoid float imprecision problems
  • Old field names removed from fields definitions. Since the scale changes by a factor or 1 million or 1 billion, we can't make them aliases with a straight face.
  • Each of these migrated fields gets a special entry in ecs-migration.yml, which mentions the scale difference between old field and event.duration, and marks them as alias: false.

@webmat webmat requested review from a team as code owners January 23, 2019 05:35
@webmat webmat self-assigned this Jan 23, 2019
@webmat webmat requested a review from ruflin January 23, 2019 05:54
@webmat
Copy link
Contributor Author

webmat commented Jan 23, 2019

@ycombinator Tweak to event.duration handling for elasticsearch.slowlog

Copy link
Contributor

@ycombinator ycombinator left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ES slowlog fileset change LGTM. Since it's a breaking change – the elasticsearch.slowlog.took_millis field is no longer being indexed — worth documenting in the breaking changes section of the CHANGELOG?

Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ruflin ruflin mentioned this pull request Jan 23, 2019
Trying to keep chattiness to a minimum by listing 2 PRs on that breaking change line.
@webmat
Copy link
Contributor Author

webmat commented Jan 23, 2019

Adding to a changelog like this: kosher?

@webmat
Copy link
Contributor Author

webmat commented Jan 23, 2019

Failing libbeat test seems unrelated to me. No field changes here could cause a problem, AFAIK.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants