Skip to content

Commit

Permalink
Convert Packetbeat Flows to ECS
Browse files Browse the repository at this point in the history 
The makes changes to the event format generated by Packetbeat's flow feature.

Field Changes

- type -> event.type
- transport -> network.transport
- flow_id -> flow.id
- final -> flow.final
- vlan -> flow.vlan
- start_time -> event.start
- last_time -> event.end
- source.stats.net_bytes_total -> source.bytes
- source.stats.net_packets_total -> source.packets
- dest.stats.net_bytes_total -> destination.bytes
- dest.stats.net_packets_total -> destination.packets

Added
- network.bytes
- network.packets
- event.duration

Frames with multiple levels of encapsulation like 802.1q with "Q-in-Q" will result in certain fields becoming an array with the outer most metadata being listed first (e.g. source.ip, destination.ip, flow.vlan).

Any dashboards associated with flows are not updated in this change.

Part of #7968.
  • Loading branch information
@andrewkroh
andrewkroh committed Dec 9, 2018
1 parent fe60901 commit 9b8c4d7
Show file tree
Hide file tree
Showing 12 changed files with 1,589 additions and 810 deletions.
2 changes: 2 additions & 0 deletions CHANGELOG.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha1...master[Check the HEAD d

*Packetbeat*

- Renamed the flow event fields to follow Elastic Common Schema. {pull}9121[9121]

*Winlogbeat*

*Functionbeat*
Expand Down
181 changes: 9 additions & 172 deletions packetbeat/_meta/fields.common.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,185 +97,22 @@
description: >
These fields contain data about the flow itself.
fields:
- name: "start_time"
type: date
required: true
format: YYYY-MM-DDTHH:MM:SS.milliZ
example: 2015-01-24T14:06:05.071Z
description: >
The time, the first packet for the flow has been seen.
- name: "last_time"
type: date
required: true
format: YYYY-MM-DDTHH:MM:SS.milliZ
example: 2015-01-24T14:06:05.071Z
description: >
The time, the most recent processed packet for the flow has been seen.
- name: final
- name: flow.final
type: boolean
description: >
Indicates if event is last event in flow. If final is false, the event
reports an intermediate flow state only.
- name: flow_id
description: >
Internal flow id based on connection meta data and address.
- name: vlan
- name: flow.id
description: >
Innermost VLAN address used in network packets.
- name: outer_vlan
description: >
Second innermost VLAN address used in network packets.
- name: source
type: group
description: >
Properties of the source host
fields:
- name: ip_location
type: geo_point
example: "40.715, -74.011"
description: >
The GeoIP location of the `ip_source` IP address. The field is a string
containing the latitude and longitude separated by a comma.
Internal flow ID based on connection meta data and address.
- name: outer_ip
description: >
Second innermost IPv4 source address as indicated by first packet seen
for the current flow.
- name: outer_ip_location
type: geo_point
example: "40.715, -74.011"
description: >
The GeoIP location of the `outer_ip_source` IP address. The field is a
string containing the latitude and longitude separated by a comma.
- name: ipv6
description: >
Innermost IPv6 source address as indicated by first packet seen for the
current flow.
- name: ipv6_location
type: geo_point
example: "60.715, -76.011"
description: >
The GeoIP location of the `ipv6_source` IP address. The field is a string
containing the latitude and longitude separated by a comma.
- name: outer_ipv6
description: >
Second innermost IPv6 source address as indicated by first packet seen
for the current flow.
- name: outer_ipv6_location
type: geo_point
example: "60.715, -76.011"
description: >
The GeoIP location of the `outer_ipv6_source` IP address. The field is a
string containing the latitude and longitude separated by a comma.
- name: stats
type: group
description: >
Object with source to destination flow measurements.
fields:
- name: net_packets_total
type: long
description: >
Total number of packets
- name: net_bytes_total
type: long
description: >
Total number of bytes
- name: dest
type: group
description: >
Properties of the destination host
fields:
- name: mac
description: >
Destination MAC address as indicated by first packet seen for the current flow.
- name: ip
description: >
Innermost IPv4 destination address as indicated by first packet seen for the
current flow.
- name: ip_location
type: geo_point
example: "40.715, -74.011"
description: >
The GeoIP location of the `ip_dest` IP address. The field is a string
containing the latitude and longitude separated by a comma.
- name: outer_ip
description: >
Second innermost IPv4 destination address as indicated by first packet
seen for the current flow.
- name: outer_ip_location
type: geo_point
example: "40.715, -74.011"
description: >
The GeoIP location of the `outer_ip_dest` IP address. The field is a
string containing the latitude and longitude separated by a comma.
- name: ipv6
description: >
Innermost IPv6 destination address as indicated by first packet seen for the
current flow.
- name: ipv6_location
type: geo_point
example: "60.715, -76.011"
description: >
The GeoIP location of the `ipv6_dest` IP address. The field is a string
containing the latitude and longitude separated by a comma.
- name: outer_ipv6
description: >
Second innermost IPv6 destination address as indicated by first packet
seen for the current flow.
- name: outer_ipv6_location
type: geo_point
example: "60.715, -76.011"
description: >
The GeoIP location of the `outer_ipv6_dest` IP address. The field is a
string containing the latitude and longitude separated by a comma.
- name: port
description: >
Destination port number as indicated by first packet seen for the current flow.
- name: stats
type: group
description: >
Object with destination to source flow measurements.
fields:
- name: net_packets_total
type: long
description: >
Total number of packets
- name: net_bytes_total
type: long
description: >
Total number of bytes
- name: icmp_id
description: >
ICMP id used in ICMP based flow.
- name: connection_id
- name: flow.vlan
type: long
description: >
optional TCP connection id
VLAN identifier from the 802.1q frame. In case of a multi-tagged frame
this field will be an array with the outer tag's VLAN identifier listed
first.
- key: trans_event
title: "Transaction Event"
Expand Down
34 changes: 34 additions & 0 deletions packetbeat/_meta/sample_outputs/flow.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
{
"@timestamp": "2018-11-15T14:41:24.000Z",
"destination": {
"bytes": 460,
"ip": "198.51.100.2",
"mac": "06:05:04:03:02:01",
"packets": 2,
"port": 80
},
"event": {
"duration": 3000000000,
"end": "2018-11-15T14:41:24.000Z",
"start": "2018-11-15T14:41:21.000Z",
"type": "flow"
},
"flow": {
"final": true,
"id": "FQQA/wz/Dv//////Fv8BAQEBAgMEBQYGBQQDAgGrAMsAcQPGM2QC9ZdQAA",
"vlan": 171
},
"network": {
"bytes": 470,
"packets": 3,
"transport": "tcp",
"type": "ipv4"
},
"source": {
"bytes": 10,
"ip": "203.0.113.3",
"mac": "01:02:03:04:05:06",
"packets": 1,
"port": 38901
}
}
4 changes: 2 additions & 2 deletions packetbeat/decoder/decoder.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -67,8 +67,8 @@ type Decoder struct {
}

const (
netPacketsTotalCounter = "net_packets_total"
netBytesTotalCounter = "net_bytes_total"
netPacketsTotalCounter = "packets"
netBytesTotalCounter = "bytes"
)

// New creates and initializes a new packet decoder.
Expand Down
Loading

0 comments on commit 9b8c4d7

Please sign in to comment.