Skip to content

Commit

Permalink
Merge branch 'develop'
Browse files Browse the repository at this point in the history
  • Loading branch information
@3c7
3c7 committed Mar 21, 2023
2 parents 32462fa + 4b7c0cd commit 7ef27d3
Show file tree
Hide file tree
Showing 3 changed files with 142 additions and 36 deletions.
132 changes: 104 additions & 28 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
While there are multiple Python projects which implement the object creation based on single VirusTotal objects, this
project aims to enable users to directly convert VirusTotal search queries to MISP objects.
**This is work in progress.** Future release will implement handling URLs, Domain and IP objects, too. Right now, only
file objects are implemented.
file objects - as a base for queries - are implemented. These file objects can have related IPs, domains and URLs, though.

## Installation

Expand All @@ -15,37 +15,113 @@ pip install vt2m

If you use the script frequently, passing the arguments as environment variables (`MISP_URL`, `MISP_KEY`, `VT_KEY`)
can be useful to save some time. For example, this can be achieved through creating a shell script which passes the
environment variables and executes the command with spaces in front, so it does not show up in the shell history.
environment variables and executes the command with spaces in front, so it does not show up in the shell history. Something like this:

```bash
#!/usr/bin env bash

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")
IFS=$SAVEIFS
MISP_URL="https://my.misp.host.local" MISP_KEY="MyMISPApiKey1234567890" VT_KEY="MyVTApiKey1234567890" /path/to/venv/bin/vt2m "$@"
IFS=$SAVEIFS
```

Changing the IFS is a must, so spaces are not seen as a field seperator.

Overall, `vt2m` supports three commands:

- VirusTotal Intelligence Search via `query`
- Accessing Live Hunting notifications via `notifications` (or `no`)
- Accessing Retrohunt results via `retrohunts` (or `re`)

### VirusTotal Ingelligence Search: `query`

```
Usage: vt2m query [OPTIONS] QUERY
Query VT for files and add them to a MISP event
Arguments:
QUERY VirusTotal Query [required]
Options:
-u, --uuid TEXT MISP event UUID [required]
-U, --url TEXT MISP URL - can be passed via MISP_URL env
-K, --key TEXT MISP API Key - can be passed via MISP_KEY env
-k, --vt-key TEXT VirusTotal API Key - can be passed via VT_KEY
env
-c, --comment TEXT Comment for new MISP objects.
-l, --limit INTEGER Limit of VirusTotal objects to receive
[default: 100]
-L, --limit-relations INTEGER Limit the amount of related objects. Note
that this is for every relation queries.
[default: 40]
-r, --relations TEXT Relations to resolve via VirusTotal,
available relations are: execution_parents,
compressed_parents, bundled_files,
dropped_files, contacted_urls, embedded_urls,
itw_urls, contacted_domains,
embedded_domains, itw_domains, contacted_ips,
embedded_ips, itw_ips, submissions,
communicating_files
-d, --detections INTEGER Amount of detections a related VirusTotal
object must at least have [default: 0]
-D, --extract-domains Extract domains from URL objects and add them
as related object.
-f, --filter TEXT Filtering related objects by matching this
string(s) against json dumps of the objects.
-p, --pivot TEXT Pivot from the given query before resolving
relationships. This must be a valid VT file
relation (execution_parents,
compressed_parents, bundled_files,
dropped_files).
-P, --pivot-limit INTEGER Limit the amount of files returned by a
pivot. [default: 40]
-C, --pivot-comment TEXT Comment to add to the initial pivot object.
--pivot-relationship TEXT MISP relationship type for the relation
between the initial pivot object and the
results. [default: related-to]
--help Show this message and exit.
```

The `query` command supports ingesting files from a VT search, but additional also requesting specific related files or infrastructure indicators (via `--relations`) and an initial pivot off the files (via `--pivot`). The latter means that, e.g., you're able to search for files that are commonly dropped or contained within the samples you're actually searching for and use the "parent" files as your regular result set, enrichting them with additional relationships etc.

Via `--relations` VirusTotal relations can be resolved and added as MISP objects with the specific relations, e.g. the
following graph was created using vt2m:
![MISP Graph](.github/screenshots/graph.png)
*Graph created via `vt2m --uuid <UUID> --limit 5 --relations dropped_files,execution_parents "behaviour_processes:\"ping -n 70\""`*

### Params
```
usage: vt2m [-h] --uuid UUID [--url URL] [--key KEY] [--vt-key VT_KEY] [--comment COMMENT] [--limit LIMIT] [--relations RELATIONS] [--quiet]
[--detections DETECTIONS]
query
positional arguments:
query VT query
optional arguments:
-h, --help show this help message and exit
--uuid UUID, -u UUID MISP event uuid
--url URL, -U URL MISP URL - can also be given as env MISP_URL
--key KEY, -k KEY MISP API key - can also be given as env MISP_KEY
--vt-key VT_KEY, -K VT_KEY
VT API key - can also be given as env VT_KEY
--comment COMMENT, -c COMMENT
Comment to add to MISP objects
--limit LIMIT, -l LIMIT
Limit results of VT query - default is 100
--relations RELATIONS, -r RELATIONS
Comma-seperated list of relations to request PER result (if type fits). This can burn your API credits. Currently
implemented: dropped_files, executing_parents, bundled_files
--quiet, -q Disable output. Stderr will still be printed.
--detections DETECTIONS, -d DETECTIONS
Only consider related entities with at least X malicious detections.
### VirusTotal Livehunt notifications: `notifications`

```
Usage: vt2m notifications [OPTIONS] COMMAND [ARGS]...
Query and process VT notifications
Options:
--help Show this message and exit.
Commands:
import Import files related to notifications
list List currently available VirusTotal notifications
```

The command allows to list and to import livehunt results via two subcommands.

### VirusTotal Retrohunt results: `retrohunts`

```
Usage: vt2m retrohunts [OPTIONS] COMMAND [ARGS]...
Query for retrohunt results.
Options:
--help Show this message and exit.
Commands:
import Imports results of a retrohunt into a MISP event
list Lists available retrohunts
```

The command allows to list and to import retrohunt results via two subcommands.
2 changes: 1 addition & 1 deletion pyproject.toml
View file
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
[tool.poetry]
name = "vt2m"
version = "0.1.9"
version = "0.1.10"
description = "Automatically import results from VirusTotal queries into MISP objects"
authors = ["3c7 <3c7@posteo.de>"]
license = "MIT"
Expand Down
44 changes: 37 additions & 7 deletions vt2m/main.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,17 +30,36 @@ def query(
"--limit-relations",
"-L",
help="Limit the amount of related objects. Note that this is for every relation queries."),
relations: str = Option("", "--relations", "-r", help=f"Relations to resolve via VirusTotal, available "
f"relations are: {', '.join(lib.all_relations)}"),
relations: str = Option(
"",
"--relations",
"-r",
help=f"Relations to resolve via VirusTotal, available relations are: {', '.join(lib.all_relations)}"
),
detections: int = Option(0, "--detections", "-d",
help="Amount of detections a related VirusTotal object must at least have"),
extract_domains: bool = Option(False, "--extract-domains", "-D",
help="Extract domains from URL objects and add them as related object."),
filter: List[str] = Option([], "--filter", "-f", help="Filtering related objects by matching this string(s) "
"against json dumps of the objects."),
pivot: str = Option(None, "-p", "--pivot", help="Pivot from the given query before resolving relationships. "
"This must be a valid VT file relation."),
pivot_limit: int = Option(40, "-P", "--pivot-limit", help="Limit the amount of files returned by a pivot.")
filter: List[str] = Option(
[],
"--filter",
"-f",
help="Filtering related objects by matching this string(s) against json dumps of the objects."
),
pivot: str = Option(
None,
"-p",
"--pivot",
help="Pivot from the given query before resolving relationships. This must be a valid VT file relation "
f"({', '.join(lib.file_relations)})."
),
pivot_limit: int = Option(40, "-P", "--pivot-limit", help="Limit the amount of files returned by a pivot."),
pivot_comment: str = Option(None, "-C", "--pivot-comment", help="Comment to add to the initial pivot object."),
pivot_relationship: str = Option(
"related-to",
"--pivot-relationship",
help="MISP relationship type for the relation between the initial pivot object and the results."
)
):
"""
Query VT for files and add them to a MISP event
Expand Down Expand Up @@ -69,6 +88,7 @@ def query(
limit=limit
)

pivot_object = None
if pivot:
pivot_results = []
for r in results:
Expand All @@ -85,6 +105,13 @@ def query(
print_err("[PIV] No files returned.")
raise typer.Exit(-1)
else:
pivot_object = lib.process_results(
results=results,
event=event,
comment=pivot_comment,
disable_output=state["quiet"],
extract_domains=False
)[0]
results = pivot_results

created_objects = lib.process_results(
Expand All @@ -105,6 +132,9 @@ def query(
limit=limit_relations,
filter=filter
)
if pivot and pivot_object:
for obj in created_objects:
lib.add_reference(pivot_object, obj.uuid, pivot_relationship)
event.published = False
misp.update_event(event)

Expand Down

0 comments on commit 7ef27d3

Please sign in to comment.