It happened to everyone -- forgotten rule in iptables
caused open access to docker
control port. Next thing you know, malicious containers running on your infrastructure, sending spam emails at best. Woudn't be great if you get notified, that something on a previously unseen TCP or UDP port started accepting connections on your server?
This project leverages an excellent masscan which can scan whole IPv4 range under 6 minutes (if you really want to push your network). We will also add support for nmap
as masscan
doesn't play nice with IPv6.
WARNING: commence port scanning from and to systems you operate, and you are allowed to send large volume of packets (e.g. TCP ACK) towards. Also make sure, that you have cleared permissions with your hosting providers. This will certainly trip various intrusion detection / anomaly detection systems.
Whole project is distributed as a Python package to make it simple to include in your tooling. It is not yet on https://pypi.org. Example Pipfile
for your project:
[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"
[packages]
masscan_as_a_service = {git = "https://github.com/bobek/masscan_as_a_service.git", editable = true}
This will give you masscan_as_a_service
binary in your bin
directory. As usual, virtualenv
/ pipenv
is recommended.
- use some system to schedule regular executions. For example gitlab CD or github actions.
- run
masscan_as_a_service masscan
, you will need- a list of targets (path passed as
--targets
). One target per line. It is up to you how you are going to generate it. For example, in our deployments, we fetch list servers to test from inventory management system (aka Machine DataBase). - access details based on your cloud provider (check below). Also prepare appropriate configuration file.
- ssh key (private and public parts) to get access to worker.
- clone repository which you use to store results and point scanner to it (
--output_dir
)
- a list of targets (path passed as
- scanner will update per host files in the output directory. You should now commit them to your repository and trigger audit in case anything changed.
Please make sure, that the provisioned VM is in the network which makes sense for your test. For example, we provision completely separete VM, which is not connected to our internal systems in any way (except control ssh
connection). Thus is represents a random Internet user as close as possible as we are interested in ports being opened to the wild.
We have mentioned use of a repository (we use git
) for storing the output files. Motivation behind it is that you will keep a history of changes within the version control. So you will know how the state looked in the past.
We also use version control as a way of detecting changes (e.g. new port being open). git
will not let you make an empty commit. On the other side, if you successfully push to remote repository with a new commit, you can trigger an action. We use Phabricator's Audits for this. Phabricator is watching repository with results and when a new commit appears, it will start a new audit. Responsible members of the team will either describe why is the new port being open as it is an intended behavior. Or they will start investigating the root cause of it being open. In any case, you have one central place for storing information about why is some port opened and accessible.
Create a project at Hetzner cloud console. Open the project and go to Access
section. Create a new API key under API Tokens
tab.
Token is then read from the environment variable, which is set in the configuration file (api_token_env
). Example configuration is at hcloud_example.yml
.
When installed, you will get a masscan_as_a_service
application. It can perform various commands based on the first positional argument.
Some arguments can be defined only on the global level. For example, you turn debugging on for cleanup
command with masscan_as_a_service -d cleanup
.
usage: masscan_as_a_service [-h] [-d] -e ENV_CONFIG {masscan,cleanup} ...
Masscan in a box
positional arguments:
{masscan,cleanup}
optional arguments:
-h, --help show this help message and exit
-d, --debug Enable debugging
-e ENV_CONFIG, --environment-config ENV_CONFIG
YAML file describing execution environment
usage: masscan_as_a_service masscan [-h] (-t TARGETS | -a API_KEYS) -o
DESTINATION_DIR --ssh-public-key
SSH_PUBLIC_KEY --ssh-private-key
SSH_PRIVATE_KEY
optional arguments:
-h, --help show this help message and exit
-t TARGETS, --targets TARGETS
File with targets (IP address) to scan. One per line.
-a API_KEYS, --api_keys API_KEYS
File with API keys of projects to scan. YAML array.
-o DESTINATION_DIR, --output_dir DESTINATION_DIR
Directory to write results to
--ssh-public-key SSH_PUBLIC_KEY
File with the public SSH key to be given access to
created VM
--ssh-private-key SSH_PRIVATE_KEY
File with the private SSH key corresponding to the
ssh-public-key
usage: masscan_as_a_service cleanup [-h] -t THRESHOLD
optional arguments:
-h, --help show this help message and exit
-t THRESHOLD, --threshold THRESHOLD
All VMs older then THRESHOLD seconds will be deleted.
I have provisioned 2 VM on Hetzner Cloud for this demo. Their IP addresses are 94.130.26.161
and 95.217.232.216
.
You would normally automate following steps in your execution system. I will perform preparation steps manually.
First, we will need some ephemeral ssh key to access worker node. We will generate a new key stored in vm_key
file with the following command:
ssh-keygen -t ed25519 -f /tmp/vm_key -q -N ""
We also need a list of targets to perform port scanning against. Let's store in the file named targets.list
.
cat <<EOF > /tmp/targets.list
94.130.26.161
95.217.232.216
EOF
Optionally you can specify list of api keys with name. masscan_as_a_service
loads targets dynamically from Hetzner.
cat <<EOF > /tmp/api_tokens.yaml
- name: jbarton # project name, description, ...
token: XfT5q...
EOF
Checkout a git repo to place results into:
git clone some_remote_repo_with_audits_enabled /tmp/out
HCLOUD_TOKEN=VERY_LONG_STRING_WITH_API_TOKEN masscan_as_a_service -d -e examples/hcloud.yml masscan --ssh-public-key /tmp/vm_key.pub --ssh-private-key /tmp/vm_key -t /tmp/targets.list -o /tmp/out
# OR
HCLOUD_TOKEN=VERY_LONG_STRING_WITH_API_TOKEN masscan_as_a_service -d -e examples/hcloud.yml masscan --ssh-public-key /tmp/vm_key.pub --ssh-private-key /tmp/vm_key -a /tmp/api_tokens.yaml -o /tmp/out
After the successful scan we will get two new files, one per target. They have the same content as on ssh
is currently open on these newly provisioned VMs.
{
"22/tcp": {
"reason": "syn-ack",
"status": "open"
}
}
We can happily commit and push our scan results. Depending on your setup, this will trigger a source code audit as we have made changes to the repository. We will just resolved it with "provisioned 2 new VMs for demoing the masscan" and move on with our DevOps work.
Scan was executed multiple times as we are running it regularly from our scheduler. But one day this happens:
diff --git i/94.130.26.161.json w/94.130.26.161.json
index bceef26..fe5c4ee 100644
--- i/94.130.26.161.json
+++ w/94.130.26.161.json
@@ -1,6 +1,26 @@
{
+ "10250/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
+ "10251/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
+ "10252/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
+ "10256/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
"22/tcp": {
"reason": "syn-ack",
"status": "open"
+ },
+ "6443/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
}
diff --git i/95.217.232.216.json w/95.217.232.216.json
index bceef26..45249f2 100644
--- i/95.217.232.216.json
+++ w/95.217.232.216.json
@@ -1,6 +1,14 @@
{
+ "10250/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
+ "10256/tcp": {
+ "reason": "syn-ack",
+ "status": "open"
+ },
"22/tcp": {
"reason": "syn-ack",
"status": "open"
}
As usual, we automatically commit and push the changed files to the repository. Source-code audit is going to be triggered as we have a new commit. We see that new ports are now open to the wild Internet. Brief look suggests that somebody has installed a new Kubernetes cluster on these VMs, but had totally forgotten to take care of filtering access to its services from the outside world.