[Packetbeat] Output ECS-compatible TLS fields #15497

adriansr · 2020-01-13T11:24:55Z

Packetbeat now outputs TLS fields from ECS 1.3+:

The additional information not covered by ECS is nested under tls.detailed.
Fields already in ECS are removed from detailed to avoid bloat.
A new configuration flag tls.include_detailed_fields allows to
toggle the inclusion of extra fields. It's enabled by default.

Caveats:

Originally it would output the top-level certificate in tls.server_certificate and the rest under tls.server_certificate_chain. ECS mandates that tls.server.certificate and tls.server.certificate_chain are mutually exclusive. To avoid confusion, a chain is always generated, even if it consists of a single certificate.
Same for tls.client certificates.
The behavior of the configuration options tls.send_certificates and tls.include_raw_certificates has slightly changed.

Non-populated TLS ECS fields:

tls.curve: Not implemented. Requires parsing the server key exchange.
tls.server.ja3s: JA3s is not implemented yet.

libbeat/beat/event.go

Packetbeat now outputs TLS fields from ECS: - The additional information not covered by ECS is nested under `tls.detailed`. - Fields already in ECS are removed from detailed to avoid bloat. - A new configuration flag `tls.include_detailed_information` allows to toggle the inclusion of extra fields. It's enabled by default. Caveats: - Originally it would output the top-level certificate in `tls.server_certificate` and the rest under `tls.server_certificate_chain`. ECS mandates that `tls.server.certificate` and `tls.server.certificate_chain` are mutually exclusive. To avoid confusion, a chain is always generated, even if it consists of a single certificate. - Same for `tls.client` certificates. Non-populated TLS ECS fields: - `tls.curve`: Requires parsing the server key exchange. - `tls.server.ja3s`: JA3s is not implemented yet.

WindowsError: [Error 206] The filename or extension is too long: 'C:\\Users\\jenkins\\workspace\\elastic+beats+pull-request+multijob-windows\\beat\\packetbeat\\label\\windows\\src\\github.com\\elastic\\beats\\packetbeat\\build\\system-tests\\run\\test_0099_golden_files.Test.test_golden_files_4_test_established_tls_not_certs_but_raw'

adriansr · 2020-01-13T16:31:01Z

Docs build are failing because I added a reference to ECS docs that needs an update in the elastic/docs repo: elastic/docs#1696

elasticmachine · 2020-01-13T17:03:18Z

Pinging @elastic/siem (Team:SIEM)

dcode

I reviewed mostly the asciidoc when I realized I probably should have reviewed the fields.yml. In some places the documentation looks like it references detailed.* fields, but the JSON examples show in the places I would expect the data to appear. Mostly, we should make sure we populate tls.client.* and tls.server.* where available. I also think the tls.client.detailed is a good approach to provide applications specific data that packetbeat can provide.

packetbeat/docs/fields.asciidoc

dcode · 2020-01-13T18:56:00Z

packetbeat/docs/fields.asciidoc

@@ -9436,7 +9447,7 @@ type: array
 The hello extensions provided by the client.


-*`tls.client_hello.extensions.server_name_indication`*::
+*`tls.detailed.client_hello.extensions.server_name_indication`*::


This should be tls.client.server_name

It is copied to the ECS field already. I just want to keep the extensions array verbatim with all the extensions present in the hello message.

dcode · 2020-01-13T18:59:35Z

packetbeat/docs/fields.asciidoc

@@ -9516,7 +9527,7 @@ type: keyword
 --


-*`tls.server_hello.version`*::
+*`tls.detailed.server_hello.version`*::


This should be parsed into tls.version for the numeric part and tls.version_protocol for the protocol name (e.g. ssl or tls)

It is already (more or less, the logic is more complicated for TLS 1.3), I just keep the raw versions from the hello message in those fields.

dcode · 2020-01-13T19:03:50Z

packetbeat/docs/fields.asciidoc

@@ -9526,7 +9537,7 @@ type: keyword

 --

-*`tls.server_hello.selected_cipher`*::
+*`tls.detailed.server_hello.selected_cipher`*::


This should be stored in tls.cipher using the IANA names, which Packetbeat already uses, I think. (see https://ciphersuite.info/cs/TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384/). @webmat maybe we should add IANA to the ECS specification so that the field data lines up?

@dcode Good idea. I opened this issue to track this elastic/ecs#720

Another field I forgot to remove.

I don't think I follow the discussion, though. Afaik ECS docs already use an IANA name as an example (TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256).

packetbeat/docs/fields.asciidoc

dcode · 2020-01-13T19:17:26Z

packetbeat/docs/fields.asciidoc

@@ -10049,7 +10096,7 @@ type: keyword

 --

-*`tls.server_certificate_chain`*::
+*`tls.detailed.server_certificate_chain`*::


how is this different than tls.server.certificate_chain? I see the code referencing both.

dcode · 2020-01-13T19:17:42Z

packetbeat/docs/fields.asciidoc

@@ -10058,7 +10105,7 @@ type: array

 --

-*`tls.client_certificate_chain`*::
+*`tls.detailed.client_certificate_chain`*::


how is this different than tls.client.certificate_chain? I see the code referencing both.

This is an array of objects, same fields as tls.detailed.client.certificate, while the ECS counterpart just stores the base64 PEM.

webmat

Thanks for getting this started, @adriansr!

Moving everything that's not ECS under tls.detailed.* will be clearer indeed. But isn't that a breaking change? Is there a mitigating factor? For example, is TLS still in beta in Packetbeat?

webmat · 2020-01-13T19:19:24Z

packetbeat/protos/tls/_meta/fields.yml

-            using a client certificate.
-
-        - name: client_hello
+        - name: detailed


Moving everything that's not ECS under tls.detailed.* will be clearer indeed. But isn't that a breaking change?

webmat · 2020-01-13T19:34:17Z

packetbeat/docs/fields.asciidoc

@@ -9526,7 +9537,7 @@ type: keyword

 --

-*`tls.server_hello.selected_cipher`*::
+*`tls.detailed.server_hello.selected_cipher`*::


@dcode Good idea. I opened this issue to track this elastic/ecs#720

andrewkroh

These changes LGTM. I like the new golden tests for Packetbeat. Since some fields were renamed (breaking change) I think we need to add aliases when we merge this into 7.x that point to the new fields (I think this is just the tls.detailed fields that will have aliases pointing to them).

libbeat/beat/event.go

Packetbeat now outputs TLS fields from ECS 1.3+: - The additional information not covered by ECS is nested under tls.detailed. - Fields already in ECS are removed from detailed to avoid bloat. - A new configuration flag tls.include_detailed_fields allows to toggle the inclusion of extra fields. It's enabled by default. Caveats: - Originally it would output the top-level certificate in tls.server_certificate and the rest under tls.server_certificate_chain. ECS mandates that tls.server.certificate and tls.server.certificate_chain are mutually exclusive. To avoid confusion, a chain is always generated, even if it consists of a single certificate. - Same for tls.client certificates. - The behavior of the configuration options tls.send_certificates and tls.include_raw_certificates has slightly changed. Non-populated TLS ECS fields: - tls.curve: Not implemented. Requires parsing the server key exchange. - tls.server.ja3s: JA3s is not implemented yet. (cherry picked from commit fd5fd3d)

…lds (#15529) Packetbeat now outputs TLS fields from ECS 1.3+: - The additional information not covered by ECS is nested under tls.detailed. - Fields already in ECS are removed from detailed to avoid bloat. - A new configuration flag tls.include_detailed_fields allows to toggle the inclusion of extra fields. It's enabled by default. Caveats: - Originally it would output the top-level certificate in tls.server_certificate and the rest under tls.server_certificate_chain. ECS mandates that tls.server.certificate and tls.server.certificate_chain are mutually exclusive. To avoid confusion, a chain is always generated, even if it consists of a single certificate. - Same for tls.client certificates. - The behavior of the configuration options tls.send_certificates and tls.include_raw_certificates has slightly changed. Non-populated TLS ECS fields: - tls.curve: Not implemented. Requires parsing the server key exchange. - tls.server.ja3s: JA3s is not implemented yet. (cherry picked from commit fd5fd3d)

adriansr added enhancement in progress Pull request is currently in progress. review Packetbeat needs_backport PR is waiting to be backported to other branches. needs_docs ecs labels Jan 13, 2020

adriansr requested a review from a team as a code owner January 13, 2020 11:24

adriansr commented Jan 13, 2020

View reviewed changes

libbeat/beat/event.go Outdated Show resolved Hide resolved

adriansr force-pushed the feature_pb_tls_ecs_14 branch from fe0be6a to 043d9ed Compare January 13, 2020 13:31

adriansr added 3 commits January 13, 2020 15:43

Add system-test that uses golden files

2c15105

Update docs

2a0b8b3

adriansr force-pushed the feature_pb_tls_ecs_14 branch from 043d9ed to 2a0b8b3 Compare January 13, 2020 14:46

adriansr removed in progress Pull request is currently in progress. needs_docs labels Jan 13, 2020

adriansr mentioned this pull request Jan 13, 2020

Update the ECS version in master to 1.4 elastic/docs#1696

Merged

adriansr added 4 commits January 13, 2020 17:08

System-test: Remove extra @metadata field

c778086

Add breaking changes to CHANGELOG

b76e070

Update dashboards

3601050

andrewkroh added the Team:SIEM label Jan 13, 2020

dcode reviewed Jan 13, 2020

View reviewed changes

webmat reviewed Jan 13, 2020

View reviewed changes

adriansr added 2 commits January 13, 2020 22:03

Enforce serialization of ECS booleans

5835c2a

Remove unused fields definitions

f63dfbf

andrewkroh approved these changes Jan 13, 2020

View reviewed changes

libbeat/beat/event.go Outdated Show resolved Hide resolved

Remove unnecessary function added to beat.Event

d46c622

adriansr merged commit fd5fd3d into elastic:master Jan 14, 2020

adriansr mentioned this pull request Jan 14, 2020

Cherry-pick #15497 to 7.x: [Packetbeat] Output ECS-compatible TLS fields #15529

Merged

adriansr added v7.6.0 and removed needs_backport PR is waiting to be backported to other branches. labels Jan 14, 2020

ycombinator mentioned this pull request Jan 14, 2020

Packetbeat setup error #15512

Closed

urso added the breaking change label Feb 6, 2020

This was referenced Mar 12, 2020

[SIEM] Non-ECS fields in SIEM elastic/kibana#43649

Open

[SIEM] Update TLS tables for ECS 1.4+ elastic/kibana#60026

Closed

ebeahan mentioned this pull request Sep 8, 2020

Description of field tls.cipher should use IANA names elastic/ecs#720

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Packetbeat] Output ECS-compatible TLS fields #15497

[Packetbeat] Output ECS-compatible TLS fields #15497

adriansr commented Jan 13, 2020 •

edited

Loading

adriansr commented Jan 13, 2020

elasticmachine commented Jan 13, 2020

dcode left a comment

dcode Jan 13, 2020

adriansr Jan 13, 2020

dcode Jan 13, 2020

adriansr Jan 13, 2020

dcode Jan 13, 2020

webmat Jan 13, 2020

adriansr Jan 13, 2020 •

edited

Loading

dcode Jan 13, 2020

dcode Jan 13, 2020

adriansr Jan 13, 2020

webmat left a comment

webmat Jan 13, 2020

webmat Jan 13, 2020

andrewkroh left a comment •

edited

Loading

[Packetbeat] Output ECS-compatible TLS fields #15497

[Packetbeat] Output ECS-compatible TLS fields #15497

Conversation

adriansr commented Jan 13, 2020 • edited Loading

adriansr commented Jan 13, 2020

elasticmachine commented Jan 13, 2020

dcode left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

adriansr Jan 13, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

webmat left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

adriansr commented Jan 13, 2020 •

edited

Loading

adriansr Jan 13, 2020 •

edited

Loading

andrewkroh left a comment •

edited

Loading