Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add eTLD+1 to the DNS documents published by Packetbeat #1107

Merged
merged 2 commits into from
Mar 8, 2016

Conversation

andrewkroh
Copy link
Member

The effective top level domain plus one more label is really useful for clustering DNS requests (hostnames). For example, the eTLD+1 for "foo.bar.golang.org." is "golang.org.".

This was the basis for the aggregations used in Detecting DNS Tunnels with Packetbeat and Watcher.

I used a Groovy script to do this in the blog post, but this has several downsides.

  • You cannot do a terms aggregations in Kibana with the eTLD+1 value because you cannot use a Groovy scripted field.
  • The method used in the blog post to get the eTLD+1 value was naive. For example, it didn't work correctly for www.google.co.uk because it would return co.uk resulting in a huge cluster of domains. This uses the data from http://publicsuffix.org so it will correctly return google.co.uk.
Unrelated changes
  • Removed non-ascii characters from the AMQP field descriptions in etc/fields.
  • Optimized the AppVeyor config. There were a lot of failures reaching out to Sourceforge to get mingw.

This change depends on the SOCKS5 PR which adds the golang.org/x/net/publicsuffix dep.

@monicasarbu
Copy link
Contributor

@andrewkroh Nice one! Is ready for review?

@andrewkroh
Copy link
Member Author

It can be reviewed, but not merged. There's a dependency on the SOCKS proxy PR which adds a common dependency. I'll add the review label as soon as it's mergeable.

@andrewkroh andrewkroh force-pushed the feature/pb-dns-etls-plus-one branch from 231c4fa to 9c4265f Compare March 8, 2016 04:53
@andrewkroh
Copy link
Member Author

This is now ready to go. I rebased it to pull in the golang.org/x/net/publicsuffix dep and added tests.

tsg added a commit that referenced this pull request Mar 8, 2016
Add eTLD+1 to the DNS documents published by Packetbeat
@tsg tsg merged commit 2006e79 into elastic:master Mar 8, 2016
@@ -6,7 +6,7 @@ This file is generated! See etc/fields.yml and scripts/generate_field_docs.py
[[exported-fields]]
== Exported Fields

This document describes the fields that are exported by Packetbeat. They are
This document describes the fields that are exported by Docs/Fields.Asciidoc. They are
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand why you changed Packetbeat to Docs/Fields.Asciidoc here.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Me either. 😕 I used make update re-build the Packetbeat docs because I added a field. I'll open a new PR to undo this change.

@andrewkroh andrewkroh deleted the feature/pb-dns-etls-plus-one branch March 8, 2016 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants