Convert Packetbeat Flows to ECS #9121

andrewkroh · 2018-11-15T23:36:29Z

The makes changes to the event format generated by Packetbeat's flow feature.

This is part of #7968.

Field Changes

type -> event.type
transport -> network.transport
flow_id -> flow.id
final -> flow.final
vlan -> flow.vlan [We might want this in ECS as network.vlan.]
start_time -> event.start
last_time -> event.end
source.stats.net_bytes_total -> source.bytes
source.stats.net_packets_total -> source.packets
dest.stats.net_bytes_total -> destination.bytes
dest.stats.net_packets_total -> destination.packets

Added

network.bytes
network.packets
event.duration

Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan).

Any dashboards associated with flows are not updated in this change.

ruflin

Could you also update the ECS migration file?

packetbeat/_meta/fields.common.yml

ruflin · 2018-11-19T08:55:16Z

packetbeat/_meta/sample_outputs/flow.json

+    "start": "2018-11-15T14:41:21.000Z",
+    "type": "flow"
+  },
+  "flow": {


I like the flow prefix.

ruflin · 2018-11-19T08:57:21Z

I wonder if we should have a second file besides ecs-migration.yml just called beats-migration.yml where we list all the fields that we migrated in Beats. This PR contains changes which are breaking but don't belong necessarily all in ecs-migration.yml.

andrewkroh · 2018-11-21T01:27:27Z

This has been updated based on the proposed changes in elastic/ecs#179.

ruflin · 2018-11-23T08:06:50Z

@andrewkroh Should in your PR description above dest.stats.net_packets_total -> destination.bytes be destination.packets? Same for source.

andrewkroh · 2018-11-27T15:00:42Z

@andrewkroh Should in your PR description above dest.stats.net_packets_total -> destination.bytes be destination.packets? Same for source.

Good catch. I fixed it in the description and the commit message.

I squashed the existing commits and rebased on master.

ruflin

I assume this PR is not planned to be backported?

We should also discuss migration compatiblity of packetbeat to 7.0 /ECS.

libbeat/_meta/fields.ecs.yml

elasticmachine · 2018-11-29T14:53:05Z

Pinging @elastic/secops

andrewkroh · 2018-11-30T22:33:11Z

I assume this PR is not planned to be backported?

@ruflin None of these fields changes are being back-ported.

ruflin · 2018-12-03T09:44:57Z

@andrewkroh As the above migrated fields are all 1-1 mappings, should we introduce aliases for it in 6.x?

andrewkroh · 2018-12-04T04:26:36Z

@ruflin Good idea. I'll added a checklist item to the parent issue for adding alias. I think it would make sense to look at the whole list of changes made for 7.0 then open a single PR to add all the alias we desire for Packetbeat to the 6.x branch.

Updating schema to commit 349406f59e5c7c80a20c9f213370d2601b73f040. Some fields were removed so I added placeholder fields with TODO statements. Those fields can be removed after modules using the fields are updated accordingly.

The makes changes to the event format generated by Packetbeat's flow feature. Field Changes - type -> event.type - transport -> network.transport - flow_id -> flow.id - final -> flow.final - vlan -> flow.vlan - start_time -> event.start - last_time -> event.end - source.stats.net_bytes_total -> source.bytes - source.stats.net_packets_total -> source.packets - dest.stats.net_bytes_total -> destination.bytes - dest.stats.net_packets_total -> destination.packets Added - network.bytes - network.packets - event.duration Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan). Any dashboards associated with flows are not updated in this change. Part of elastic#7968.

andrewkroh · 2018-12-10T11:25:33Z

@ruflin I rebased this since it’s been in existence for a while. Can you please take another look?

What changed?

Pulled in the latest copy of ECS. (first commit)
Any fields that were removed from ECS that we still used elsewhere were moved to the bottom of fields.ecs.yml with a comment.
Nothing changed on the Packetbeat side (the second commit).

ruflin

LGTM. @webmat Perhaps you could also have a look?

ruflin · 2018-12-11T09:36:00Z

packetbeat/tests/system/test_0060_flows.py

@@ -34,22 +34,22 @@ def test_mysql_flow(self):
        pprint(objs)


What is this new pcap file just above (can't comment there)? Is that intentional in this PR?

ruflin · 2018-12-11T09:37:33Z

libbeat/_meta/fields.ecs.yml

@@ -97,6 +118,125 @@
            different values which are then freely searchable. If multiple
            messages exist, they can be combined into one message.

+    - name: client


I'm not fully convinced we should add the full ecs fields yml to each Beat or only add the fields which are used by more then one Beat. We can still clean this up later.

exekias · 2019-01-09T15:53:13Z

libbeat/_meta/fields.ecs.yml

          type: keyword
          description: >
            Unique ID to describe the event.
          example: 8a4f500d
-          phase: 1
+
+        - name: kind


this backport failed because kind is not present in 6.x: #9962, do you plan to backport this PR?

Alternatively, should I copy back this file to 6.x? cc @ruflin

The Packetbeat flow changes are not being back-ported to 6.x.

* Update ECS fields Updating schema to commit 349406f59e5c7c80a20c9f213370d2601b73f040. Some fields were removed so I added placeholder fields with TODO statements. Those fields can be removed after modules using the fields are updated accordingly. * Convert Packetbeat Flows to ECS The makes changes to the event format generated by Packetbeat's flow feature. Field Changes - type -> event.type - transport -> network.transport - flow_id -> flow.id - final -> flow.final - vlan -> flow.vlan - start_time -> event.start - last_time -> event.end - source.stats.net_bytes_total -> source.bytes - source.stats.net_packets_total -> source.packets - dest.stats.net_bytes_total -> destination.bytes - dest.stats.net_packets_total -> destination.packets Added - network.bytes - network.packets - event.duration Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan). Any dashboards associated with flows are not updated in this change. Part of elastic#7968.

andrewkroh added in progress Pull request is currently in progress. review Packetbeat ecs labels Nov 15, 2018

ruflin mentioned this pull request Nov 16, 2018

Transition Beats to ECS #8655

Closed

ruflin reviewed Nov 19, 2018

View reviewed changes

andrewkroh force-pushed the feature/pb/ecs-flows branch from 29676e0 to 5fee60a Compare November 21, 2018 01:40

andrewkroh mentioned this pull request Nov 21, 2018

[Packetbeat] - Change field names to follow ECS #7968

Closed

9 tasks

andrewkroh force-pushed the feature/pb/ecs-flows branch from d8db504 to ace9b93 Compare November 27, 2018 14:59

andrewkroh removed the in progress Pull request is currently in progress. label Nov 27, 2018

andrewkroh force-pushed the feature/pb/ecs-flows branch from ace9b93 to 6e87f68 Compare November 28, 2018 02:26

ruflin reviewed Nov 29, 2018

View reviewed changes

libbeat/_meta/fields.ecs.yml Show resolved Hide resolved

libbeat/_meta/fields.ecs.yml Outdated Show resolved Hide resolved

andrewkroh added the SecOps label Nov 29, 2018

andrewkroh mentioned this pull request Nov 30, 2018

[Packetbeat] Restructure client/server and process fields #9303

Merged

andrewkroh mentioned this pull request Dec 4, 2018

Proposal for more straightforward network metrics fields. elastic/ecs#179

Merged

andrewkroh force-pushed the feature/pb/ecs-flows branch from 6e87f68 to 8c2fc44 Compare December 8, 2018 17:52

andrewkroh added 2 commits December 8, 2018 23:12

Update ECS fields

fe60901

Updating schema to commit 349406f59e5c7c80a20c9f213370d2601b73f040. Some fields were removed so I added placeholder fields with TODO statements. Those fields can be removed after modules using the fields are updated accordingly.

andrewkroh force-pushed the feature/pb/ecs-flows branch from 25514e9 to 9b8c4d7 Compare December 9, 2018 05:21

andrewkroh requested a review from ruflin December 10, 2018 11:22

ruflin approved these changes Dec 11, 2018

View reviewed changes

andrewkroh merged commit 66a9525 into elastic:master Dec 12, 2018

andrewvc mentioned this pull request Dec 12, 2018

Add geo fields to add_host_metadata processor. #9392

Merged

cwurm mentioned this pull request Dec 18, 2018

Add Auditbeat system module fields to fields.ecs.yml #9318

Closed

exekias reviewed Jan 9, 2019

View reviewed changes

andrewkroh mentioned this pull request Jan 15, 2019

Update flows to use event.dataset #10063

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert Packetbeat Flows to ECS #9121

Convert Packetbeat Flows to ECS #9121

andrewkroh commented Nov 15, 2018 •

edited

Loading

ruflin left a comment

ruflin Nov 19, 2018

ruflin commented Nov 19, 2018

andrewkroh commented Nov 21, 2018

ruflin commented Nov 23, 2018

andrewkroh commented Nov 27, 2018

ruflin left a comment

elasticmachine commented Nov 29, 2018

andrewkroh commented Nov 30, 2018

ruflin commented Dec 3, 2018

andrewkroh commented Dec 4, 2018

andrewkroh commented Dec 10, 2018

ruflin left a comment

ruflin Dec 11, 2018

ruflin Dec 11, 2018

exekias Jan 9, 2019

andrewkroh Jan 10, 2019

Convert Packetbeat Flows to ECS #9121

Convert Packetbeat Flows to ECS #9121

Conversation

andrewkroh commented Nov 15, 2018 • edited Loading

ruflin left a comment

Choose a reason for hiding this comment

ruflin Nov 19, 2018

Choose a reason for hiding this comment

ruflin commented Nov 19, 2018

andrewkroh commented Nov 21, 2018

ruflin commented Nov 23, 2018

andrewkroh commented Nov 27, 2018

ruflin left a comment

Choose a reason for hiding this comment

elasticmachine commented Nov 29, 2018

andrewkroh commented Nov 30, 2018

ruflin commented Dec 3, 2018

andrewkroh commented Dec 4, 2018

andrewkroh commented Dec 10, 2018

ruflin left a comment

Choose a reason for hiding this comment

ruflin Dec 11, 2018

Choose a reason for hiding this comment

ruflin Dec 11, 2018

Choose a reason for hiding this comment

exekias Jan 9, 2019

Choose a reason for hiding this comment

andrewkroh Jan 10, 2019

Choose a reason for hiding this comment

andrewkroh commented Nov 15, 2018 •

edited

Loading