Threat Bus 2020.11.26

We’re happy to announce the monthly release 2020.11.26 of Threat Bus.

Threat Bus — Performance

This release brings lots of performance tweaks. Additionally, we now release pyvast-threatbus as individual PyPI package and maintain a separate changelog.

Plugin Performance

Plugins in Threat Bus work with queues to deliver threat intelligence. Application plugins, such as the zmq-app plugin, create one queue for each new subscribing app. These queues are filled by the backbone with exactly those messages that match the subscribed topics. For example, pyvast-threatbus subscribes to the topic threatbus/intel via the zmq-app plugin. The plugin creates a new queue and passes it to the backbone. When the backbone puts new intel into that queue, the zmq-app plugin takes it and forwards it to pyvast-threatbus.

Backbones and application plugins are decoupled. It’s the application plugins’ responsibility to check for new messages and forward them to the subscribing app, e.g., using protocols like ZeroMQ or Zeek’s Broker.

The old Threat Bus implementation did not scale well with our usual workloads. In our setup, we maintain threat intelligence in MISP and other tools and store historical network telemetry in VAST. We use Threat Bus with the MISP and zmq-app plugins to ultimately forward intelligence updates from MISP to VAST. With the retro-matching feature of pyvast-threatbus we check new IoCs against historical data stored in VAST to effectively uncover past incidents with new threat intelligence in the fastest way possible.

We found that message consumption and intelligence matching was significantly slower than our IoC update. Threat Bus stacked up a growing backlog of undelivered messages, because our TI tools published IoCs way faster than the zmq-app plugin would deliver them to pyvast-threatbus.

We quickly identified the bottleneck. Application plugins, like the Zeek and zmq-app plugin, used a busy-polling approach to check subscriber queues for new messages from the backbone. Prior to this release, Threat Bus used the standard Python3 queues. Unfortunately, that implementation lacks performant notification schemes like select or epoll.

To fix this, we had to change the type of queues that Threat Bus uses internally. Starting this release, Threat Bus now uses joinable queues. They are backed by native OS primitives and can be selected or epolled.

Both the Zeek and zmq-app plugins now use select to publish new messages to subscribers the very moment the backbone shoves them into the subscriber’s queue. We opted for select to keep the logic simple, portable, and sufficiently performant to no longer be the bottleneck. The figure below shows the performance boost for message publishing in the zmq-app plugin.

We also improved the message consumption speed of the RabbitMQ backbone plugin. Instead of a synchronous, blocking connection, we now use an asynchronous select connection that works entirely based on callbacks. The following graphic shows the speedup of the RabbitMQ consumer in the backbone plugin.

Note that the second figure has a much smaller Y-axis than the first. To highlight the impact of the changes, we plotted them all together in the next graphic.

In addition to changing the internally used queue type, we introduced a new base class to implement asynchronous background threads in Threat Bus plugins. All existing plugins now extend that class to implement their workers. Using that base class allows us to implement a graceful application shutdown. When the user presses CTRL+c to terminate Threat Bus, it will first shut down backbone plugins, then application plugins, and lastly Threat Bus itself.

PyVAST-ThreatBus

This release also brings a great performance boost to pyvast-threatbus, the application that enables VAST to connect with Threat Bus. pyvast-threatbus consumes threat intelligence from Threat Bus, transforms it to valid VAST queries, and pipes back the query results in the form of sightings. Prior to this release we used to work sequentially through all incoming IoCs. Now, pyvast-threatbus works with asynchronous background tasks and can issue hundreds of concurrent VAST queries at once. Users can configure the maximum amount of concurrent background tasks in the config file.

Changelog Highlights

Starting with this release, pyvast-threatbus comes with its own changelog. We will release it monthly together with Threat Bus. The changelog highlights cover both Threat Bus and pyvast-threatbus.

🎁 Features

• Pressing ctrl+c shuts down Threat Bus and its plugins gracefully.
  #61
• The Zeek and zmq-app plugins now use the select system call to provision messages to subscribers instead of busy polling. This increases performance by an order of magnitude, while it keeps compatibility with all supported OSes.
  #61
• The zmq-app plugin now supports heart beats. This enables both Threat Bus and connected apps to mutually ensure that the connected party is still alive.
  #58
• pyvast-threatbus can now issue multiple concurrent VAST queries instead of working through all received IoCs sequentially. Users can configure the maximum number of concurrent background tasks in the configuration file.
  #61

⚠️ Changes

• The new StoppableWorker base class should be implemented by all Plugins for their asynchronous background tasks. All existing plugins in the official Threat Bus repository now implement this class.
  #61

🐞 Bugs

• pyvast-threatbus escapes backslashes and quotes in IoCs before it queries VAST.
  #74