Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Drop pkt_dstaddr and pkt_srcaddr when equals to "-" #22721

Merged
merged 5 commits into from
Nov 30, 2020
Merged

Drop pkt_dstaddr and pkt_srcaddr when equals to "-" #22721

merged 5 commits into from
Nov 30, 2020

Conversation

kaiyan-sheng
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng commented Nov 23, 2020

What does this PR do?

This PR is to fix parsing error for vpcflow fileset when aws.vpcflow.pkt_dstaddr or aws.vpcflow.pkt_srcaddr equals to -.

Error message:

failed to parse field [aws.vpcflow.pkt_srcaddr] of type [ip] in document with id '229a9b7009-000001896666'. Preview of field's value: '-'

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 23, 2020
@kaiyan-sheng kaiyan-sheng self-assigned this Nov 23, 2020
@kaiyan-sheng kaiyan-sheng added review Team:Platforms Label for the Integrations - Platforms team labels Nov 23, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 23, 2020
@kaiyan-sheng kaiyan-sheng added the needs_backport PR is waiting to be backported to other branches. label Nov 23, 2020
@elasticmachine
Copy link
Collaborator

elasticmachine commented Nov 23, 2020

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Started by user kaiyan-sheng

  • Start Time: 2020-11-30T14:10:15.748+0000

  • Duration: 46 min 40 sec

Test stats 🧪

Test Results
Failed 0
Passed 1979
Skipped 259
Total 2238

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 1979
Skipped 259
Total 2238

Copy link
Contributor

@leehinman leehinman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

pipeline changes look good.

Any idea why geo.country_name is removed from the expected.json?

@kaiyan-sheng
Copy link
Contributor Author

pipeline changes look good.

Thank you for your help!!

Any idea why geo.country_name is removed from the expected.json?

For some reason I have to manually change elasticsearch snapshot version from 8.0.0 to 7.10.0 in order to keep the geo.country_name. I regenerated the files and let's see if CI is happy this time.

@kaiyan-sheng kaiyan-sheng merged commit 24a4da8 into elastic:master Nov 30, 2020
@kaiyan-sheng kaiyan-sheng deleted the vpcflow branch November 30, 2020 15:22
@ChrsMark ChrsMark added v7.11.0 and removed needs_backport PR is waiting to be backported to other branches. labels Dec 1, 2020
ChrsMark pushed a commit to ChrsMark/beats that referenced this pull request Dec 1, 2020
* Add painless script to remove all empty fields

(cherry picked from commit 24a4da8)
ChrsMark pushed a commit to ChrsMark/beats that referenced this pull request Dec 1, 2020
* Add painless script to remove all empty fields

(cherry picked from commit 24a4da8)
kaiyan-sheng added a commit that referenced this pull request Dec 1, 2020
…als to "-" (#22825)

* Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (#22721)

* Add painless script to remove all empty fields

(cherry picked from commit 24a4da8)

* fix changelog

Signed-off-by: chrismark <chrismarkou92@gmail.com>

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
ChrsMark added a commit that referenced this pull request Dec 2, 2020
* Add painless script to remove all empty fields

(cherry picked from commit 24a4da8)

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
v1v added a commit to v1v/beats that referenced this pull request Dec 2, 2020
…-issues

* upstream/master: (41 commits)
  Fix version parser regex for packaging (elastic#22581)
  Fix local_dynamic documentation and add providers inline doc. (elastic#22657)
  fix: use proper param name for e2e tests (elastic#22836)
  [Heartbeat] Fix exit on disabled monitor (elastic#22829)
  Update Golang to 1.14.12 (elastic#22790)
  docs: fix setup.template.overwrite typos (elastic#22804)
  Add docs section for ECS EC2 monitoring (elastic#22784)
  Fixing logic to keep list of unique cluster UUIDs (elastic#22808)
  Skip somewhat flaky UDP system test on Windows (elastic#22810)
  Fix polling node when it is not ready and monitor by hostname (elastic#22666)
  Skip Filebeat test_shutdown on windows 7 (elastic#22797)
  Make monitoring Namespace thread-safe (elastic#22640)
  Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721)
  Add support for reading from UNIX datagram sockets (elastic#22699)
  Fix export dashboard command from Elastic Cloud (elastic#22746)
  Skip flaky winlogbeat test on Windows-7 (elastic#22754)
  Missing `>` (elastic#22763) (elastic#22766)
  Fix k8s watcher issue when node access to list nodes and ns (elastic#22714)
  [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732)
  Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634)
  ...
v1v added a commit to v1v/beats that referenced this pull request Dec 2, 2020
…dows-7

* upstream/master: (41 commits)
  Fix version parser regex for packaging (elastic#22581)
  Fix local_dynamic documentation and add providers inline doc. (elastic#22657)
  fix: use proper param name for e2e tests (elastic#22836)
  [Heartbeat] Fix exit on disabled monitor (elastic#22829)
  Update Golang to 1.14.12 (elastic#22790)
  docs: fix setup.template.overwrite typos (elastic#22804)
  Add docs section for ECS EC2 monitoring (elastic#22784)
  Fixing logic to keep list of unique cluster UUIDs (elastic#22808)
  Skip somewhat flaky UDP system test on Windows (elastic#22810)
  Fix polling node when it is not ready and monitor by hostname (elastic#22666)
  Skip Filebeat test_shutdown on windows 7 (elastic#22797)
  Make monitoring Namespace thread-safe (elastic#22640)
  Drop pkt_dstaddr and pkt_srcaddr when equals to "-" (elastic#22721)
  Add support for reading from UNIX datagram sockets (elastic#22699)
  Fix export dashboard command from Elastic Cloud (elastic#22746)
  Skip flaky winlogbeat test on Windows-7 (elastic#22754)
  Missing `>` (elastic#22763) (elastic#22766)
  Fix k8s watcher issue when node access to list nodes and ns (elastic#22714)
  [Metricbeat/Kibana/stats] Enforce `exclude_usage=true` (elastic#22732)
  Avoid sending non-numeric floats in cloud foundry integrations (elastic#22634)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
review Team:Platforms Label for the Integrations - Platforms team v7.10.1 v7.11.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[filebeat][aws][vpcflow] mapping errors on logs with "NODATA" & "SKIPDATA" log_status
4 participants