Merge pull request #99

Use optional dependencies for different MISP integrations

CHANGELOG.md

@@ -12,6 +12,15 @@ Every entry has a category for which we use the following visual abbreviations:
 
 ## Unreleased
 
+- 🎁 Feature
+  The MISP plugin now uses [extra dependencies](https://www.python.org/dev/peps/pep-0508/#extras).
+  Users can now chose the wanted dependencies during installation by running
+  `pip install threatbus-misp[zmq]` to install the ZeroMQ dependency, or
+  `pip install threatbus-misp[kafka]` to install the Kafka dependency. The
+  plugin throws a fatal error if none of these dependencies is installed and
+  exits immediately.
+  [#99](https://github.com/tenzir/threatbus/pull/99)
+
 - 🎁 Feature
   The RabbitMQ backbone plugin, the In-memory backbone plugins, and the Zmq-app
   plugin now support the

plugins/apps/threatbus_misp/Makefile

@@ -30,8 +30,8 @@ dist:
 
 .PHONY: install
 install:
-	pip install .
+	pip install .[zmq]
 
 .PHONY: dev-mode
 dev-mode:
-	pip install --editable .
+	pip install --editable .[zmq]

plugins/apps/threatbus_misp/README.md

@@ -9,32 +9,59 @@ Threat Bus MISP Plugin
 
 </h4>
 
-A Threat Bus plugin that enables communication to [MISP](https://www.misp-project.org/).
+A Threat Bus plugin that enables communication with [MISP](https://www.misp-project.org/).
 
 The plugin goes against the pub/sub architecture of Threat Bus (for now),
-because the plugin subscribes a listener to ZeroMQ / Kafka, rather than having
-MISP subscribe itself to Threat Bus. That will be addressed with a MISP module
-in the near future.
+because it actively binds to a single MISP instance to receive attribute
+(IoC) updates, and report back sightings via the REST API. Following the strict
+pub/sub architecture of Threat Bus, it *should be the other way
+around*, with MISP binding to Threat Bus. This will eventually be resolved by a
+MISP module.
+
+For now, the plugin supports two ways to retrieve attribute (IoC) updates from
+MISP - either via ZeroMQ or via Kafka. Basically, the plugin makes itself a
+subscriber to MISP events.
 
 ## Installation
 
+Users can specify *optional dependencies* during installation. The plugin uses
+either ZeroMQ or Kafka to get IoC updates from MISP. As we don't want to burden
+the user to install unused dependencies, both options are available as follows:
+
+
 ```sh
-pip install threatbus-misp
+pip install threatbus-misp[zmq]
+pip install threatbus-misp[kafka]
 ```
 
-#### Prerequisites
+If neither of these dependencies is installed (i.e., you installed
+`threatbus-misp` without the `[...]` suffix for optional deps), the plugin throws
+an error and exits immediately.
+
+**Depending on your setup, you might want to use quotes to avoid shell expansion
+when using `[...]`**. For example, you can do `pip install ".[zmq]"` for local
+development.
 
-*Install Kafka on the Threat Bus host*
+### Kafka Prerequisites
 
-The plugin enables communication either via ZeroMQ or Kafka. When using Kafka,
-you have to install `librdkafka` for the host system that is running
-`threatbus`. See also the [prerequisites](https://github.com/confluentinc/confluent-kafka-python#prerequisites)
-section of the `confluent-kafka` python client.
+When you decide to use Kafka to receive IoC updates from MISP, you first need to
+install Kafka on the Threat Bus host. This plugin uses the
+[confluent-kafka](https://docs.confluent.io/platform/current/clients/confluent-kafka-python/index.html)
+Python package which requires `librdkafka`. See also the
+[prerequisites](https://github.com/confluentinc/confluent-kafka-python#prerequisites)
+section of the `confluent-kafka` Python client for details about setting it up
+for your distribution.
+
+Once installed, go ahead and install the Kafka version of this plugin:
+
+```
+pip install threatbus-misp[kafka]
+```
 
 ## Configuration
 
-The plugin can either use ZeroMQ or Kafka to retrieve intelligence items from
-MISP. It uses the MISP REST api to report back sightings of indicators.
+The plugin uses the MISP REST API to report back sightings of IoCs. You need to
+specify a MISP API key for it to work.
 
 ZeroMQ and Kafka are mutually exclusive, such that Threat Bus does not receive
 all attribute updates twice. See below for an example configuration.
@@ -79,7 +106,7 @@ plugins:
 ...
 ```
 
-### Filter
+### IoC Filter
 
 The plugin can be configured with a list of filters. Every filter describes a
 whitelist for MISP attributes (IoCs). The MISP plugin will only forward IoCs to
@@ -219,7 +246,7 @@ misp-server:
 make deploy
 ```
 
-*Enable the Kafka plugin in the MISP webview*
+*Enable the Kafka plugin in the MISP web-view*
 
 - Visit https://localhost:80
 - login with your configured credentials
@@ -250,7 +277,7 @@ service apache2 restart
 exit # leave the Docker container shell
 ```
 
-*Enable the ZMQ plugin in the MISP webview*
+*Enable the ZMQ plugin in the MISP web-view*
 
 - Visit https://localhost:80
 - login with your configured credentials

plugins/apps/threatbus_misp/setup.py

@@ -26,11 +26,10 @@
     description="A plugin to enable threatbus communication with MISP.",
     entry_points={"threatbus.app": ["misp = threatbus_misp.plugin"]},
     install_requires=[
-        "threatbus>=2020.12.16",
-        "pymisp >= 2.4.120, <= 2.4.134",
-        "pyzmq>=18.1.1",
-        "confluent-kafka>=1.3.0",
+        "threatbus >= 2020.12.16, < 2021.2.24",
+        "pymisp >= 2.4.120",
     ],
+    extras_require={"kafka": ["confluent-kafka>=1.3.0"], "zmq": ["pyzmq>=18.1.1"]},
     keywords=[
         "MISP",
         "zeromq",

plugins/apps/threatbus_misp/threatbus_misp/plugin.py

@@ -1,4 +1,15 @@
-from confluent_kafka import Consumer
+## The MISP plugin lists two optional dependencies via `extras_requires`, so
+## users can either install `threatbus-misp[zmq]` or `threatbus-misp[kafka]`.
+## The plugin needs at least one of these to work properly, so we we need to log
+## an error if none of these is installed.
+dep_kafka = False
+dep_zmq = False
+try:
+    from confluent_kafka import Consumer
+
+    dep_kafka = True
+except ModuleNotFoundError:
+    pass
 from confuse import Subview
 from datetime import datetime
 from itertools import product
@@ -12,7 +23,13 @@
 from threatbus_misp.message_mapping import map_to_internal, map_to_misp, is_whitelisted
 from typing import Callable, List, Dict
 import warnings
-import zmq
+
+try:
+    import zmq
+
+    dep_zmq = True
+except ModuleNotFoundError:
+    pass
 
 
 warnings.simplefilter("ignore")  # pymisp produces urllib warnings
@@ -162,9 +179,15 @@ def validate_config(config: Subview):
         config["api"]["ssl"].get(bool)
         config["api"]["key"].get(str)
     if config["zmq"].get(dict):
+        assert (
+            dep_zmq
+        ), "MISP attribute export is configured via ZeroMQ, but the dependency is not installed. Install `threatbus-misp[zmq]` to use this setting."
         config["zmq"]["host"].get(str)
         config["zmq"]["port"].get(int)
     if config["kafka"].get(dict):
+        assert (
+            dep_kafka
+        ), "MISP attribute export is configured via Apache Kafka, but the dependency is not installed. Install `threatbus-misp[kafka]` to use this setting."
         config["kafka"]["topics"].get(list)
         config["kafka"]["poll_interval"].add(1.0)
         config["kafka"]["poll_interval"].get(float)