Skip to content

VAST 2020.04.29
Compare
Choose a tag to compare
@tobim tobim released this 29 Apr 19:49
2020.04.29
This tag was signed with the committer’s verified signature.
tobim
GPG key ID: F8657E90819A1298
Learn about vigilant mode.
df8e4c5
This commit was created on github.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

We are happy to announce our release 2020.04.29. Similar to last month’s release, this release mainly included bug and robustness fixes, along with adding more documentation over at docs.tenzir.com.

  • Data Sets. we have begun to add experimental support for data sets, a deterministic abstraction for pinning a query result as a working set. Because they are deterministic, they support paging, sorting, and other blocking operations. For now, we prototype the API in Python. We may move the implementation back to C++ at some later point if we encounter performance issues. Data Sets will be fully implemented on top of Apache Arrow so that they can benefit from zero-copy data sharing across multiple processes. VAST will structurally share it’s data with the Data Set Manager in a copy-on-write fashion. For example, if 80% of all data is shared among 10 queries, this will result in 800% memory reduction. Data Sets will also be the intermediate data representation for our upcoming web UI. Stay tuned.
  • Archive performance. Requests to the archive are now interruptible, reducing the latency of exports for large databases especially. First results now appear quicker for queries matching many results, and the overall export performance has improved.
  • IoC Matching. We reworked the user interface to the intelligence matching feature: you can use the new vast matcher start subcommand to start a new matcher and get a stream of matches on the standard output. Multiple matchers can run in parallel this way, each of which operates on a subset of the ingested data. See the documentation for more details.

Improvements

  • 🎁 Bash autocompletion for vast is now available via the autocomplete script located at scripts/vast-completions.bash in the VAST source tree. #833

  • 🎁 Packet drop and discard statistics are now reported to the accountant for PCAP import, and are available using the keys pcap-reader.recv, pcap-reader.drop, pcap-reader.ifdrop, pcap-reader.discard, and pcap-reader.discard-rate in the vast.statistics event. If the number of dropped packets exceeds a configurable threshold, VAST additionally warns about packet drops on the command line. #827 #844

Improvements (Pro Only)

  • 🎁 Added a new subcommand vast matcher start. It is now possible to create multiple matchers running in parallel using the new subcommand, each of which can be configured to match a subset of the input stream. The old --enable-matcher flag was removed.

  • 🎁 Matchers can now load existing IoCs from VAST on startup using the --ioc-query flag.

  • 🎁 Matchers now support live removal of single IoCs using the vast matcher remove-ioc subcommand.

Changes

  • 🔄 The option --skip-candidate-checks / -s for the count command was renamed to --estimate / -e. #843

  • 🔄 The index specific options --max-partition-size, --max-resident-partitions, --max-taste-partitions, and --max-queries can now be specified on the command line when starting a node. #728

  • 🔄 The default bind address has been changed from :: to localhost. #828

Bug Fixes

  • 🪲 For some queries, the index evaluated only a subset of all relevant partitions in a non-deterministic manner. Fixing a violated evaluation invariant now guarantees deterministic execution. #842

  • 🪲 The stop command always returned immediately, regardless of whether it succeeded. It now blocks until the remote node shut down properly or returns an error exit code upon failure. #849

  • 🪲 Fixed a crash when importing data while a continuous export was running for unrelated events. #830

  • 🪲 Fixed a bug that could cause stalled input streams not to forward events to the index and archive components for the JSON, CSV, and Syslog readers, when the input stopped arriving but no EOF was sent. This is a follow-up to #750. A timeout now ensures that the readers continue when some events were already handled, but the input appears to be stalled. #835

  • 🪲 Queries of the form x != 80/tcp were falsely evaluated as x != 80/? && x != ?/tcp. (The syntax in the second predicate does not yet exist; it only illustrates the bug.) Port inequality queries now correctly evaluate x != 80/? || x != ?/tcp. E.g., the result now contains values like 80/udp and 80/?, but also 8080/tcp. #834

  • 🪲 Archive lookups are now interruptible. This change fixes an issue that caused consecutive exports to slow down the node, which improves the overall performance for larger databases considerably. #825

As always, see the CHANGELOG for a full list of changes.

Assets 4
Loading