Network Flow Query Language (NFQL) is our in-house stream-based
flow-query language. The query domain specific langauge (DSL) language
allows to describe patterns in flow records in a declarative and
orthogonal fashion.
- Filter flows.
- Combine flows into groups.
- Aggregate flows on flow-keys as one grouped flow aggregate.
- Merge grouped flows, supporting temporal relations between groups.
- Apply absolute or relative filters when grouping or merging.
- Unfold grouped flows back into individual flows.
nfql is a reference implementation of NFQL. nfql is composed of an
execution engine and a query parser. The execution engine is the brain
of nfql where the flows are processed. nfql reads the flow-query in
an intermediate JSON format and reads in the trace files in memory for
efficient processing. The query parser can be used to read a custom
query DSL and generate the JSON intermediate representation required
by the execution engine.
- Supports reading and writing NetFlow v5 flows in
flow-toolsformat. - The default query DSL can be mathematically expressed as a DNF expression.
- The query is read in a
JSONformat allowingM2Mcommunication. - The query parser and execution engine are decoupled from one another.
- The flow processing is modular adhering to a five-stage processing pipeline.
- Each stage of the processing pipeline can be enabled/disabled at runtime.
- Compression levels for flow-traces can be configured at runtime.
The JSON representation need not be manually generated by the user. A
NFQL parser must read, syntactically check and parse the query DSL to
generate an equivalent JSON representation. Currently we are missing
such a parser module. For now, we provide custom Python scripts that
can generate JSON representations. A new query and its corresponding
unit test can be prepared by adapting to one of the respective
example/ and tests/ scripts provided in the package.
Tried on Debian Wheezy x86_64
Install Dependencies
$ sudo apt-get install cmake flow-tools-dev zlib1g-dev libjson0-dev
Install IPFIX protocol library
$ sudo apt-get install libglib2.0-dev
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install
Build and install nfql
[nfql] $ make
[nfql] $ sudo make install
$ sudo ldconfig
Read the man page
$ sudo apt-get install most
$ man nfql | most
Generate Documentation (optional)
$ sudo apt-get install doxygen
$ sudo apt-get install graphviz
[nfql] $ make doc
Cleanup
[nfql] $ make clean
Tried on Mac OS X 10.9.1
Install Homebrew →
$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
Install Dependencies
$ brew install cmake
$ brew install flow-tools
NOTE: json-c starting from v0.11 has renamed the library from
libjson to libjson-c. Debian-based systems are currently providing
v0.10 in the repository. In order to avoid complexity in the makefile,
we have decided to freeze the library dependency to v0.10. Install
json-c v0.10:
$ brew install https://raw.github.com/mxcl/homebrew/9f78662acff4c3cb1bff3587c8ead7f5935925a9/Library/Formula/json-c.rb
Install IPFIX protocol library
$ brew install libfixbuf
$ brew install gettext
$ brew link --force gettext
Build and install nfql
[nfql] $ make
[nfql] $ make install
Read the man page
$ brew install most
$ man nfql | most
Generate Documentation (optional)
$ brew install doxygen
$ brew install graphviz
[nfql] $ make doc
Cleanup
[nfql] $ make clean
WARNING: This documentation is obsolete.
Tried on Mac OS X 10.7.
Install MacPorts →
Install Dependencies:
$ sudo port install cmake flow-tools json-c
Build nfql:
[nfql] $ make CMAKE_PREFIX_PATH=/opt/local
[nfql] $ make install
Read the man page:
$ man nfql
Generate Documentation (optional)
$ sudo port install doxygen graphviz
[nfql] $ make doc
Cleanup
[nfql] $ make clean
Tried on Fedora 18.
Install Dependencies
$ sudo yum install cmake flow-tools-devel json-c-devel
Install IPFIX protocol library
$ sudo yum install glib2-devel
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install
Build and install nfql
[nfql] $ make
[nfql] $ sudo make install
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig
Read the man page
$ sudo yum install most
$ man nfql | most
Generate Documentation (optional)
$ sudo yum install doxygen graphviz
[nfql] $ make doc
Cleanup
[nfql] $ make clean
WARNING: This documentation is obsolete.
Tried on FreeBSD 9.2
Install Dependencies
$ cd /usr/ports/devel/cmake
[cmake] $ sudo make install
$ cd /usr/ports/net-mgmt/flow-tools
[flow-tools] $ sudo make install
$ cd /usr/ports/devel/json-c
[json-c] $ sudo make install
$ cd /usr/ports/devel/glib20
[glib20] $ sudo make install
$ cd /usr/ports/net/libfixbuf
[libfixbuf] $ sudo make install
$ cd /usr/ports/devel/libexecinfo
[libexecinfo] $ sudo make install
Build and install nfql
[nfql] $ make CMAKE_PREFIX_PATH=/usr/local
[nfql] $ make install
Read the man page
$ cd /usr/ports/sysutils/most
[most] $ sudo make install
$ man nfql | most
Generate Documentation (optional)
$ cd /usr/ports/devel/doxygen
[doxygen] $ sudo make install
$ cd /usr/ports/graphics/graphviz
[graphviz] $ sudo make install
[nfql] $ make doc
Cleanup
[nfql] $ make clean
Some example queries are provided in examples/ along with a sample trace.
[nfql] $ nfql --ipfix examples/query-http-tcp-session.json examples/trace-2009.ipfix
[nfql] $ nfql examples/query-http-tcp-session.json examples/trace-2009.ft
The sample queries can also be run on your own NetFlow v5 records
[nfql] $ flow-cat tracefile[s] | nfql examples/query-http-tcp-session.json
Requirements: Python 2.7+
To run the complete regression test-suite:
[nfql] $ tests/regression.py [-v]
Regression tests can also be run individually on a specific example query type. For instance:
[nfql] $ tests/test-query-http-tcp-session.py [-v]
Requirements: Python 2.7+
To run the nfql benchmarks:
[nfql] $ make
[nfql] $ sudo benchmarks/nfql.py nfql tracefile[s]/ querie[s]/
Example nfql traces and queries are provided in examples/
To run the SiLK benchmarks:
[nfql] $ sudo benchmarks/silk.py tracefile[s]/ querie[s]/
Example SiLK traces and queries are provided in examples/silk/
- Vaibhav Bajpai contact@vaibhavbajpai.com
- Johannes Schauer j.schauer@email.de
- Corneliu Claudiu Prodescu c.prodescu@jacobs-university.de
- Jürgen Schönwälder j.schoenwaelder@jacobs-university.de