Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix _id field in s3 and googlepubsub inputs #17026

Merged

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Mar 16, 2020

What does this PR do?

In #15859 the Elasticsearch output was changed to read from the @metadata._id field when it had been using @metadata.id. The s3 and googlepubsub inputs had both been setting @metadata.id, but were not updated with that change.

This updates the s3 and googlepubsub inputs to use beat.Event#SetID() rather than creating the metadata object themselves.

Why is it important?

These inputs rely on the Elasticsearch _id field to prevent duplicate events from being written to Elasticsearch in certain edge cases.

Related issues

@andrewkroh andrewkroh added bug Filebeat Filebeat needs_backport PR is waiting to be backported to other branches. labels Mar 16, 2020
In elastic#15859 the Elasticsearch output was changed to read from the @metadata._id field when it had been using @metadata.id.
The s3 and googlepubsub inputs had both been setting @metadata.id, but were not updated with that change.

This updates the s3 and googlepubsub inputs to use `beat.Event#SetID()` rather than creating the metadata object themselves.
@andrewkroh andrewkroh force-pushed the bugfix/filebeat-input-es-id-field branch from cf754bf to c656da4 Compare March 16, 2020 15:06
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks for fixing the s3 input part!

@andrewkroh andrewkroh merged commit 304eca4 into elastic:master Mar 18, 2020
@andresrc andresrc added [zube]: Done Team:Platforms Label for the Integrations - Platforms team labels Mar 19, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

andrewkroh added a commit to andrewkroh/beats that referenced this pull request Mar 19, 2020
In elastic#15859 the Elasticsearch output was changed to read from the @metadata._id field when it had been using @metadata.id.
The s3 and googlepubsub inputs had both been setting @metadata.id, but were not updated with that change.

This updates the s3 and googlepubsub inputs to use `beat.Event#SetID()` rather than creating the metadata object themselves.

(cherry picked from commit 304eca4)
@andrewkroh andrewkroh added v7.7.0 and removed needs_backport PR is waiting to be backported to other branches. labels Mar 19, 2020
andrewkroh added a commit that referenced this pull request Mar 19, 2020
In #15859 the Elasticsearch output was changed to read from the @metadata._id field when it had been using @metadata.id.
The s3 and googlepubsub inputs had both been setting @metadata.id, but were not updated with that change.

This updates the s3 and googlepubsub inputs to use `beat.Event#SetID()` rather than creating the metadata object themselves.

(cherry picked from commit 304eca4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Filebeat Filebeat Team:Platforms Label for the Integrations - Platforms team v7.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants