Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update "Custom Brimcap Config" wiki article #340

Merged
merged 3 commits into from
Apr 16, 2024
Merged

Conversation

philrz
Copy link
Contributor

@philrz philrz commented Apr 11, 2024

Now that the build-zeek repo brings us up to date with Zeek, I wanted to give this article in the Brimcap wiki a test run to see if anything needed to be changed. I ended up fixing and improving a handful of things. I'll put in-line comments with explanations for several changes.

Comment on lines 31 to 32
and [Suricata](https://suricata.readthedocs.io/en/latest/install.html#install-binary-packages).
We'll use Linux Ubuntu 20.04 as our OS platform. On such a host, the following
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strictly speaking, we can't say that GA releases of both Zeek and Suricata are newer than the ones with Brimcap, since we're now up-to-date on Zeek. Therefore I dropped that part of the text.

Comment on lines 59 to 63
2. Other changes are made to the default configuration of the Brimcap-bundled
Zeek, such as enabling
[Community ID Flow Hashing](https://docs.zeek.org/en/master/customizations.html#community-id).
See the [build-zeek release automation](https://github.com/brimdata/build-zeek/blob/main/.github/workflows/release.yml)
for details on how this and other customizations are handled.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We used to use a Zeek package for Community ID, but now it's included out-of-the-box with Zeek so I've been taking advantage of that in the Zeek builds that come out of the new build-zeek repo. This gave me a nice excuse to link to the build-zeek repo, since it's a better starting place than what we had before for users that want to try their hand at making their own custom Zeek builds, including on Windows.

can also be pointed at the path to this configuration file, which will cause it
to be invoked when you open or drag pcap files into Zui.

![Zui YAML Config File Preference](media/Zui-Pref-YAML-Config-File.png)
![Zui YAML Config File Setting](media/Zui-Settings-YAML-Config-File.png)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Zui menu used to be called "Preferences" in some OSes and "Settings" on others, but thankfully we've standardized on "Settings" across the board now.

#!/bin/bash
exec /opt/zeek/bin/zeek -C -r - --exec "event zeek_init() { Log::disable_stream(PacketFilter::LOG); Log::disable_stream(LoadedScripts::LOG); }" local
exec /opt/zeek/bin/zeek -C -r - --exec "event zeek_init() { Log::disable_stream(PacketFilter::LOG); Log::disable_stream(LoadedScripts::LOG); Log::disable_stream(Telemetry::LOG); }" local
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The newer Zeek releases have these telemetry logs that are suited primarily to gathering perf info about running deployments (e.g., live capture environments) but they still get generated when processing pcaps. Their volume dwarfs the amount of actual analyzed events when small pcaps are processed, and it seems doubtful their contents would be essential viewing for the pcap use case. I've been excluding them via the Zeek Runners that are bundled with the builds from the new build-zeek repo, so I do the same here.

@@ -306,12 +309,12 @@ export LD_LIBRARY_PATH="/usr/local/lib"

As we did with Zeek and Suricata, we create a [wrapper script](https://github.com/brimdata/brimcap/blob/main/examples/nfdump-wrapper.sh) to act as our
Brimcap analyzer. It works in two phases, first creating binary NetFlow records
and then converting them to CSV. `nfpcapd` only accepts a true pcap file input
and then converting them to NDJSON. `nfpcapd` only accepts a true pcap file input
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I ran the latest nfdump today for the first time in a while, it seems they recently mucked up their CSV support (they now output one of their fields onto a separate line, which made Zed's CSV reader choke), but the good news is that they added an NDJSON output format that I'm more than happy to recommend instead. I'd only been using nfdump's CSV output because their JSON output was always a giant array in the past which meant bumping into brimdata/super#3865. Now that they can output NDJSON (which they call json-log since it's more compatible with log-centric tools like Splunk and Logstash) we can take advantage of that.

@philrz philrz marked this pull request as ready for review April 11, 2024 19:22
@philrz philrz self-assigned this Apr 11, 2024
@philrz philrz requested review from mattnibs, nwt and jameskerr April 11, 2024 19:22
docs/Custom-Brimcap-Config.md Outdated Show resolved Hide resolved
docs/Custom-Brimcap-Config.md Outdated Show resolved Hide resolved
philrz and others added 2 commits April 16, 2024 11:35
Co-authored-by: Noah Treuhaft <noah.treuhaft@gmail.com>
Co-authored-by: Noah Treuhaft <noah.treuhaft@gmail.com>
@philrz philrz merged commit 1884fc1 into main Apr 16, 2024
4 checks passed
@philrz philrz deleted the update-custom-article branch April 16, 2024 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants