Skip to content

Commit

Permalink
H3ll0 w0rld!
Browse files Browse the repository at this point in the history
  • Loading branch information
@cmpxchg16
cmpxchg16 committed Mar 3, 2021
1 parent f31ccfe commit 5cdd9c1
Showing 1 changed file with 26 additions and 26 deletions.
52 changes: 26 additions & 26 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,23 +3,23 @@

# netz :globe_with_meridians::eagle:

The purpose of this project is to discover an internet wide misconfiguration of network components like web-servers/databases/cache-services and more.
The purpose of this project is to discover an internet-wide misconfiguration of network components like web-servers/databases/cache-services and more.
The basic use-case for such misconfiguration - a service that is publicly exposed to the world without a credentials `¯\_(ツ)_/¯`

You probably familiar with tools like [Shodan](https://www.shodan.io/), [Censys](https://censys.io/), [ZoomEye](https://www.zoomeye.org/) to query such wide internet components,
but here we are going to do it in the fun way :: by hands :D
but here we are going to do it in a fun way :: by hands :D

The tools we are going to use are [masscan](https://github.com/robertdavidgraham/masscan), and [zgrab2](https://github.com/zmap/zgrab2) from [ZMap](https://zmap.io/) project. For the first phase of port scanning we will use [masscan](https://github.com/robertdavidgraham/masscan), then for the second phase we will run [zgrab2](https://github.com/zmap/zgrab2) to check applicative access for those ports.
The tools we are going to use are [masscan](https://github.com/robertdavidgraham/masscan), and [zgrab2](https://github.com/zmap/zgrab2) from [ZMap](https://zmap.io/) project. For the first phase of port scanning, we will use [masscan](https://github.com/robertdavidgraham/masscan), then for the second phase, we will run [zgrab2](https://github.com/zmap/zgrab2) to check applicative access for those ports.

[ZMap](https://github.com/zmap/zmap) is also internet wide scanner, so why [masscan](https://github.com/robertdavidgraham/masscan) and not [ZMap](https://github.com/zmap/zmap)..?
[ZMap](https://github.com/zmap/zmap) is also internet-wide scanner, so why [masscan](https://github.com/robertdavidgraham/masscan) and not [ZMap](https://github.com/zmap/zmap)..?
because we want to go wild and use kernel module [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) to get blazing fast packets-per-second to scan the entire internet in minutes,
and [ZMap](https://github.com/zmap/zmap) basically does support it in the past, but now [ZMap](https://github.com/zmap/zmap) doesn't compatible with the latest [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/).

Note that [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) requires license per MAC/NIC (you can run 5 minutes in demo before it will kill the flow), and you need special NIC from Intel (don't worry, the public cloud has such) so you can go without this module, and pay on time to wait for results.
Note that [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) requires a license per MAC/NIC (you can run 5 minutes in demo before it will kill the flow), and you need a special NIC from Intel (don't worry, the public cloud has such) so you can go without this module, and pay on time to wait for results.

There are few options to run this project:

1. Use netz cloud runner tool - this tool automate the full pipeline including infrastructure on top of AWS
1. Use netz cloud runner tool - this tool automate the full pipeline, including infrastructure on top of AWS
2. Run by yourself using docker
3. For [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) run by yourself the infrastructure and using [pf_ring setup](pf_ring/configure_pf_ring.sh)

Expand All @@ -33,14 +33,14 @@ The flow is:
* pipe ip list from step 1 into [zgrab2](https://github.com/zmap/zgrab2) (you can change with `ZGRAB2_ENDPOINT` environment variable for any [Elasticsearch](https://www.elastic.co/) API Endpoint, for instance: `/_cat/indices`
* extract with [jq](https://stedolan.github.io/jq/) just those ip's that return HTTP 200 OK and include `lucene_version`

The result of this flow are ip's that has an internet access to [Elasticsearch](https://www.elastic.co/) without credentials.
This flow result is ips' that has internet access to [Elasticsearch](https://www.elastic.co/) without credentials.

This test flow demonstrates [Elasticsearch](https://www.elastic.co/) scan, you can run such scans on any port (service port) you wish and on any supported protocol by [zgrab2 modules](https://github.com/zmap/zgrab2/tree/master/modules). To control such different scan options, there are environment variables you can configure:
This test flow demonstrates [Elasticsearch](https://www.elastic.co/) scan. You can run such scans on any port (service port) you wish and on any supported protocol by [zgrab2 modules](https://github.com/zmap/zgrab2/tree/master/modules). Environment variables can modify more control:
`PORT_TO_SCAN`
`SUBNET_TO_SCAN`
`ZGRAB2_ENDPOINT`

Incase you wish to add a missing protocol, you can extend [zgrab2](https://github.com/zmap/zgrab2) by [adding new protocols](https://github.com/zmap/zgrab2#adding-new-protocols.)
In case you wish to add a missing protocol, you can extend [zgrab2](https://github.com/zmap/zgrab2) by [adding new protocols](https://github.com/zmap/zgrab2#adding-new-protocols.)


We will go through a setup to be faster and faster (decreasing the time to wait).
Expand All @@ -56,11 +56,11 @@ What it does:
* Associate IAM role to Instance Profile
* Create Temporary ECS Cluster
* Create EC2 instance (instance type based on user input `--instance-type`)
* Create number of Network Interfaces (number based on user input `--number-of-nic`)
* Create a number of Network Interfaces (number based on user input `--number-of-nic`)
* Create Public Elastic IP (number based on user input `--number-of-nic`)
* Associate Elastic IP with Network Interface (for each user input `--number-of-nic`)
* Run ECS task with the scan pipeline
* Create CloudWatch log group and stream the pipeline docker output into user terminal
* Run ECS task with the scanning pipeline
* Create CloudWatch log group and stream the pipeline docker output into the user terminal
* Destroying all AWS resources
* Done

Expand Down Expand Up @@ -113,12 +113,12 @@ $ netz --file taskdefinition.json --security-group sg-XXXXXXXXXXXXXXXXXX --subne
:warning:
**Because masscan meltdown the network, SSH mostly will not be available, also CloudWatch logs will be deferred, so the tailed logs in user terminal will take some time.**

Note that [taskdefinition.json](taskdefinition.json) is related to running with the automate way with AWS ECS.
In that file you will be able to change the subnet & port to scan, also the application endpoint as well.
In this file you can also control the CPU & RAM you allocate to the task, for the test we assumed you run it with c4.8xlarge, so the config is `60 x cpu` and `36 GB RAM`.
Note that [taskdefinition.json](taskdefinition.json) is related to running with the automated way with AWS ECS.
In that file, you will be able to change the subnet & port to scan, also the application endpoint.
In this file, you can also control the CPU & RAM you allocate to the task. This test assumed c4.8xlarge, so the config is `60 x cpu` and `36 GB RAM`.

### Result
We tested it on AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS => took 25 minutes
On AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS => took 25 minutes

## 2. Run by yourself using docker

Expand All @@ -132,21 +132,21 @@ $ docker build -t netz .
$ docker run -e PORT_TO_SCAN='9200' -e SUBNET_TO_SCAN='0.0.0.0/0' -e ZGRAB2_ENDPOINT='/' -e TASK_DEFINITION='docker' -v /tmp/:/opt/out --network=host -it netz
```
:warning:
**The time to scrape the entire internet with simple hardware and simple internet backbone could takes days**
**The time to scrape the entire internet with simple hardware and simple internet backbone could take days**


### 3. Faster :zap:
#### Run with Docker on Cloud with one 10gbps NIC

Run instance with one 10gbps NIC (e.g in AWS c4.8xlarge [already configured with])
Run instance with one 10gbps NIC (e.g. in AWS c4.8xlarge [already configured with])

Steps are the same as [2.1 Basic](https://github.com/SpectralOps/netz#21-basic).

### Result
We tested it on AWS with c4.8xlarge ~ 700k ~ 950k PPS => took 2.5 hours
On AWS with c4.8xlarge ~ 700k ~ 950k PPS => took 2.5 hours.

### 4. Faster++ :zap::dizzy:
#### Run with Docker on Cloud with multiple 10gbps NIC (e.g in AWS c4.8xlarge 10gbps NIC )
#### Run with Docker on Cloud with multiple 10gbps NIC (e.g. in AWS c4.8xlarge 10gbps NIC )
* Run in AWS c4.8xlarge Ubuntu 18.04 and connect multiple NIC ([ENI's](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html))
* For each NIC you need to configure the OS to see those new NIC's.

Expand Down Expand Up @@ -223,17 +223,17 @@ adapter-mac[2] = 06:YY:YY:YY:YY:YY
```

### Result
We tested it on AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS => took 35 minutes
On AWS with c4.8xlarge with 6 x NIC ~ 2.9M ~ 3.5M PPS => took 35 minutes


### 5. Faster++++ :zap::dizzy::tornado:
#### Run on Cloud with 10gbps NIC with [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/)
In case you want to scrape the internet in a few minutes with [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/), you will need to run a machine that supports the kernel device drivers, and a machine that has 10gbps NIC.
In case you want to scrape the internet in a few minutes with [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/), you will need to run a machine that supports the kernel device drivers and a machine that has 10gbps NIC.

Notes:
* Because [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) bypasses the TCP stack, so in case you have just one NIC ens3 and you will open it with **zc**:enc3, you will lose SSH access. If you still want SSH access, you will need another NIC, e.g ens4, then open ens4 with **zc**, so it will be **zc**:ens4, so ens3 will continue as management NIC for SSH.
* If you will run machine with 1gbps NIC, it will still be fast, but it will take **x10** more time you could `¯\_(ツ)_/¯`
* You don't have to run such machine like c4.8xlarge, you can run each machine that support the ixgbevf
* Because [PF_RING ZC (Zero Copy)](https://www.ntop.org/products/packet-capture/pf_ring/pf_ring-zc-zero-copy/) bypasses the TCP stack, so in case you have just one NIC ens3 and you will open it with **zc**:enc3, you will lose SSH access. If you still want SSH access, you will need another NIC, e.g. ens4, then open ens4 with **zc**, so it will be **zc**:ens4, so ens3 will continue as management NIC for SSH.
* If you run a machine with 1gbps NIC, it will still be fast, but it will take **x10** more time you could `¯\_(ツ)_/¯`
* You don't have to run such a machine like c4.8xlarge, you can run each machine that supports the ixgbevf
from: [enhanced networking with the Intel 82599 VF interface](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sriov-networking.html)


Expand Down Expand Up @@ -288,7 +288,7 @@ PORT_TO_SCAN='9200' SUBNET_TO_SCAN='0.0.0.0/0' ZGRAB2_ENDPOINT='/' TASK_DEFINITI


### Result
We tested it on AWS with c4.8xlarge with 4 x NIC ~ 10.5M ~ 12M PPS => took 10 minutes
On AWS with c4.8xlarge with 4 x NIC ~ 10.5M ~ 12M PPS => took 10 minutes


# Disclaimer
Expand Down

0 comments on commit 5cdd9c1

Please sign in to comment.